Oct 30 19:03:24.375412 ip-10-0-132-19 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 19:03:24.375421 ip-10-0-132-19 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 19:03:24.375428 ip-10-0-132-19 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 19:03:24.375669 ip-10-0-132-19 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 19:03:34.528965 ip-10-0-132-19 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 19:03:34.528983 ip-10-0-132-19 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 408b09930a4f45678c6fb83d1d0420b8 --
Oct 30 19:05:45.000917 ip-10-0-132-19 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:05:45.383651 ip-10-0-132-19 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:05:45.383651 ip-10-0-132-19 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:05:45.383651 ip-10-0-132-19 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:05:45.383651 ip-10-0-132-19 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:05:45.388257 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388165    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:45.388257 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388252    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:45.388257 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388257    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:45.388257 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388261    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:45.388257 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388264    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388269    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388272    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388277    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388280    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388287    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388293    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388298    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388302    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388306    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388310    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388314    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388318    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388321    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388325    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388329    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388333    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388338    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388342    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:45.388550 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388346    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388350    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388360    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388364    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388368    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388372    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388376    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388380    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388384    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388388    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388393    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388397    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388401    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388405    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388409    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388414    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388419    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388423    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388427    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:45.389283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388431    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388436    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388441    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388447    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388451    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388455    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388460    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388464    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388469    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388473    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388477    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388481    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388486    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388490    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388494    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388498    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388502    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388506    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388512    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388516    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:45.389790 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388521    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388525    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388530    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388534    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388538    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388542    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388546    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388550    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388554    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388559    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388562    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388566    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388570    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388575    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388579    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388583    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388587    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388592    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388596    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388601    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:45.390578 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388605    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388609    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388614    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388619    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388623    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388627    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388654    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388661    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388666    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388670    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388674    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388679    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388683    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388687    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388691    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388695    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388699    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388703    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388707    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388711    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:45.391451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388717    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388721    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388726    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.388730    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390839    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390851    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390856    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390861    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390865    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390871    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390875    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390880    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390885    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390889    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390894    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390902    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390910    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390915    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390921    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390927    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:45.392270 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390932    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390936    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390940    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390944    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390949    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390953    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390957    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390961    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390965    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390970    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390974    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390979    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390984    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390988    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390993    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.390997    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391001    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391006    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391010    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:45.393143 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391015    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391019    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391024    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391028    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391032    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391038    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391044    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391050    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391056    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391060    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391064    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391069    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391073    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391077    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391081    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391085    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391090    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391094    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391099    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:45.393950 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391103    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391107    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391111    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391117    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391121    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391125    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391130    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391135    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391140    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391144    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391148    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391152    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391156    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391161    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391164    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391169    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391173    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391177    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391181    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391187    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:45.394840 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391191    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391195    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391200    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391205    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391209    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391213    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391217    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391222    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391226    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391231    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391235    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391239    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391243    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391247    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391251    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391255    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391260    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391266    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391270    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:45.395431 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391275    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391279    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391283    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391287    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391291    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391295    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391300    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391304    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391309    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391313    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391317    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391322    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.391327    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392212    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392223    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392229    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392234    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392239    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392243    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:45.396146 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392248    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392252    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392256    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392261    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392265    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392269    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392273    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392277    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392281    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392285    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392289    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392293    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392298    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392303    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392307    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392312    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392316    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392320    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392324    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392328    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:45.396593 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392333    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392337    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392341    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392345    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392349    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392353    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392358    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392362    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392366    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392371    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392375    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392379    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392384    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392387    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392392    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392396    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392400    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392404    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392408    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392412    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:45.397123 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392416    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392420    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392425    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392428    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392433    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392437    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392441    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392445    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392449    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392453    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392457    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392461    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392465    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392472    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392478    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392483    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392487    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392491    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392495    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:45.397612 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392500    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392504    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392508    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392514    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392518    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392524    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392528    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392532    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392536    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392540    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392544    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392548    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392555    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392561    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392565    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392569    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392574    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392578    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392582    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392587    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:45.398070 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392592    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392596    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392600    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392604    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392608    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392612    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392616    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392620    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392624    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392628    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392632    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392651    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392656    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392660    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392664    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392668    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392674    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392678    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392682    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392686    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:45.398536 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.392691    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392826    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392838    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392847    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392855    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392862    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392867    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392874    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392880    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392886    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392891    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392896    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392902    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392907    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392912    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392917    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392921    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392926    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392931    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392938    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392943    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392948    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392954    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392959    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:05:45.399070 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392965    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392969    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392975    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392980    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392985    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392991    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.392995    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393000    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393005    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393011    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393016    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393021    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393025    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393030    2578 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393035    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393041    2578 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393049    2578 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393056    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393061    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393065    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393072    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393077    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393082    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393086    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393091    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:05:45.399629 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393095    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393100    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393105    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393109    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393114    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393119    2578 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393126    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393131    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393136    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393140    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393144    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393149    2578 flags.go:64] FLAG: --help="false"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393153    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393158    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393163    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393167    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393173    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393178    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393183    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393187    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393192    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393197    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393202    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:05:45.400369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393209    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393216    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393221    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393226    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393230    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393235    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393240    2578 flags.go:64] FLAG: --lock-file=""
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393244    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393249    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393254    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393262    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393267    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393272    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393276    2578 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393281    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393286    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393291    2578 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393296    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393303    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393308    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393313    2578 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393318    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393323    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393328    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393333    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:05:45.400918 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393338    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393343    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393347    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393358    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393363    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393368    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393373    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393378    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393384    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393389    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393393    2578 flags.go:64] FLAG: --port="10250"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393398    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393403    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0833cd20562798d9c"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393408    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393413    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393418    2578 flags.go:64] FLAG: --register-node="true"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393423    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393429    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393433    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393438    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393442    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393448    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393453    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393457    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393462    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393467    2578 flags.go:64] FLAG: --runonce="false"
Oct 30 19:05:45.401486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393472    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393476    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393481    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393486    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393490    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393496    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393501    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393506    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393510    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393514    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393519    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393523    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393528    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393535    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393541    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393549    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393554    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393558    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393565    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393570    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393574    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393579    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393584    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393588    2578 flags.go:64] FLAG: --v="2"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393595    2578 flags.go:64] FLAG: --version="false"
Oct 30 19:05:45.402096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393601    2578 flags.go:64] FLAG: --vmodule=""
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393607    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.393612    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393855    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393864    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393871    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393878    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393884    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393888    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393892    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393896    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393900    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393904    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393909    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393914    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393918    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393923    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393927    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393931    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:45.402761 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393935    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393942    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393947    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393951    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393955    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393959    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393963    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393967    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393971    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393975    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393979    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393983    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393987    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393992    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.393996    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394000    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394004    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394008    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394012    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394017    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:45.403198 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394021    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394025    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394029    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394034    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394038    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394042    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394046    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394051    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394055    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394059    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394063    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394068    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394072    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394078    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394084    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394090    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394094    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394098    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394102    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394107    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:45.403697 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394111    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394116    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394120    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394124    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394128    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394132    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394136    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394140    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394144    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394149    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394153    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394157    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394161    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394165    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394169    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394174    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394178    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394182    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394186    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394190    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:45.404199 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394195    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394199    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394203    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394207    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394211    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394216    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394222    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394226    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394230    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394234    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394238    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394242    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394246    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394250    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394254    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394258    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394262    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394266    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394271    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:45.404657 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394275    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394281    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394286    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394290    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394294    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394298    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394302    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394306    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394310    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394314    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394318    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.394365    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394429    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394437    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394442    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394447    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:45.405090 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394451    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394455    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394459    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394468    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394472    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394476    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394480    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394485    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394489    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394493    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394497    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394501    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394505    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394509    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394514    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394518    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394529    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394534    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394538    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:45.405454 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394542    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394546    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394551    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394555    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394576    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394582    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394588    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394592    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394597    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394602    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394606    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394610    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394615    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394620    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394624    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394628    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394653    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394660    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:45.405903 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394666    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394671    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394676    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394680    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394684    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394688    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394693    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394697    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394701    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394706    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394710    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394716    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394720    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394724    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394729    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394733    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394737    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394742    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394746    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394750    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:45.406312 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394754    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394758    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394762    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394767    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394771    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394775    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394780    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394785    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394790    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394794    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394801    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394805    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394810    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394815    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394819    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394823    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394827    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394831    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394835    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394839    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:45.406816 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394843    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394847    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394851    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394855    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394859    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394862    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394867    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394871    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394875    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394880    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394884    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394888    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394892    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394897    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394902    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394907    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394911    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394915    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394919    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394924    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:45.407283 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394929    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394933    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394939    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394944    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.394948    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.401584    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.401601    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.402322    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.402330    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402384    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402390    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402393    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402397    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402400    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402402    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402405    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402408    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402410    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402413    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402415    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:45.407785 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402418    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402420    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402425    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402429    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402433    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402436    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402438    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402441    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402443    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402446    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402449    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402452    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402455    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402457    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402460    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402462    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402465    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402468    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402471    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:45.408239 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402474    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402477    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402480    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402482    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402485    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402488    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402490    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402493    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402495    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402498    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402501    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402503    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402506    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402508    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402511    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402513    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402516    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402519    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402521    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402531    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:45.408690 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402534    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402537    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402539    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402542    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402545    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402548    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402551    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402553    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402556    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402558    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402561    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402564    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402567    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402570    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402573    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402575    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402596    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402599    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402602    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:45.409169 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402605    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402607    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402610    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402612    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402615    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402617    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402620    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402623    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402626    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402629    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402631    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402644    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402647    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402650    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402654    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402658    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402661    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402663    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402666    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:45.409655 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402669    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402671    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402675    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402677    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402680    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402683    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402686    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402690    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402693    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402696    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402698    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402701    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402703    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402706    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402709    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402711    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402714    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:45.410086 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402716    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.402745    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402887    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402891    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402895    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402898    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402901    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402903    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402906    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402909    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402912    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402914    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402917    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402920    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402923    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402925    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:45.410478 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402928    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402931    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402933    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402936    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402938    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402941    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402944    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402947    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402950    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402953    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402955    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402958    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402960    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402963    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402965    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402968    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402970    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402973    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402975    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402978    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:45.410896 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402981    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402983    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402986    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402989    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402991    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402993    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402996    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.402998    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403001    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403004    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403006    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403009    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403012    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403014    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403017    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403019    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403022    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403024    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403027    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403030    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:45.411360 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403033    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403036    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403038    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403040    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403043    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403045    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403048    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403050    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403053    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403055    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403058    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403060    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403063    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403065    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403068    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403070    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403073    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403075    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403078    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403081    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:45.411952 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403084    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403087    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403089    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403092    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403094    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403098    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403101    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403105    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403108    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403112    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403116    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403119    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403121    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403125    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403128    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403130    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403133    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403136    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403139    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:45.412395 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403141    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403144    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403146    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403149    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403151    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403154    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403156    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403159    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403161    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403164    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403166    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403169    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403172    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.403194    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403226    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403230    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:45.412839 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403234    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403236    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403239    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403241    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403244    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403246    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403249    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403251    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403254    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403256    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403259    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403262    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403265    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403267    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403270    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403272    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403275    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403277    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403279    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:45.413189 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403282    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403286    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403290    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403293    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403297    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403300    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403303    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403306    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403309    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403312    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403314    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403317    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403320    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403323    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403325    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403328    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403330    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403333    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403335    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:45.413611 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403337    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403340    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403342    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403345    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403347    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403349    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403353    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403355    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403358    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403360    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403362    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403365    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403368    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403371    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403373    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403379    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403382    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403384    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403387    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:45.414047 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403389    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403392    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403395    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403397    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403399    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403402    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403405    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403407    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403410    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403412    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403414    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403417    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403420    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403422    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403425    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403427    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403430    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403432    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403435    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:45.414463 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403437    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403440    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403443    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403447    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403449    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403452    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403454    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403457    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403460    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403463    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403466    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403468    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403471    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403474    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403477    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403479    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403482    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403484    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403487    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403490    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:45.414937 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403493    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403495    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403498    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403501    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403504    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403506    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403509    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:45.403511    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.403563    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.405442    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.406346    2578 server.go:1014] "Starting client certificate rotation"
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.406438    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.406476    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.408196    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:05:45.415400 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.411606    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:05:45.423147 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.423126    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:05:45.426186 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.426174    2578 log.go:25] "Validated CRI v1 image API"
Oct 30 19:05:45.427071 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.427040    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:05:45.431450 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.431429    2578 fs.go:135] Filesystem UUIDs: map[07aeb59a-2716-45de-b149-709b77101aa5:/dev/nvme0n1p3 408a1e8f-02d5-4175-be67-f3a77e4a2f89:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:05:45.431533 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.431449    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:05:45.437966 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.437852    2578 manager.go:217] Machine: {Timestamp:2026-10-30 19:05:45.436574464 +0000 UTC m=+0.320457658 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3185438 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d129208b79bad133d6d7d185a1516 SystemUUID:ec2d1292-08b7-9bad-133d-6d7d185a1516 BootID:408b0993-0a4f-4567-8c6f-b83d1d0420b8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:90:8b:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:90:8b:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:b9:34:36:fe:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:05:45.438438 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.438428    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:05:45.438603 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.438589    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:05:45.440800 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.440774    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:05:45.440947 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.440801    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-19.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:05:45.440991 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.440949    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:05:45.440991 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.440959    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:05:45.441039 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.440991    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:05:45.441039 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.441004    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:05:45.441757 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.441747    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:05:45.444504 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.444485    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:05:45.444566 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.444556    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:05:45.444712 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.444702    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:05:45.444747 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.444740    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:05:45.444781 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.444756    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:05:45.444781 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.444767    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:05:45.444829 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.444800    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:05:45.444829 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.444809    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:05:45.445174 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.445162    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:05:45.447248 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.447233    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:05:45.451825 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.451812    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:05:45.452683 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.452670    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:05:45.452726 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.452701    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:05:45.452879 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.452868    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:05:45.452920 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.452885    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:05:45.452920 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.452892    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:05:45.452920 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.452897    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:05:45.452920 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.452904    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:05:45.452920 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.452909    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:05:45.453448 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.453440    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:05:45.453476 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.453458    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:05:45.453476 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.453465    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:05:45.453476 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.453471    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:05:45.453657 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.453483    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:05:45.453657 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.453493    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:05:45.454181 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.454171    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:05:45.454226 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.454185    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:05:45.458165 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.458153    2578 server.go:1263] "Started kubelet"
Oct 30 19:05:45.458250 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.458226    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:05:45.458292 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.458238    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:05:45.458320 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.458304    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:05:45.459034 ip-10-0-132-19 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:05:45.459424 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.459409    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:05:45.465255 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.465166    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwf8n"
Oct 30 19:05:45.466128 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.466109    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:05:45.466746 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.466721    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:05:45.469162 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.469142    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:05:45.469595 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.469581    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:05:45.469968 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.469921    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:45.470126 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470107    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:05:45.470212 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470110    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:05:45.470212 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470147    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:05:45.470301 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470258    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:05:45.470301 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470267    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:05:45.470392 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470337    2578 factory.go:55] Registering systemd factory
Oct 30 19:05:45.470392 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470351    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:05:45.470588 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470555    2578 factory.go:153] Registering CRI-O factory
Oct 30 19:05:45.470588 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470588    2578 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:05:45.470744 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470631    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:05:45.470744 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470664    2578 factory.go:103] Registering Raw factory
Oct 30 19:05:45.470744 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.470677    2578 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:05:45.471509 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.471497    2578 manager.go:319] Starting recovery of all containers
Oct 30 19:05:45.477286 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.477254    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-19.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:05:45.477559 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.477539    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-19.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:05:45.477731 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.476746    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-19.us-east-2.compute.internal.18e3641b65c797ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-19.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-19.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:05:45.458063276 +0000 UTC m=+0.341946466,LastTimestamp:2026-10-30 19:05:45.458063276 +0000 UTC m=+0.341946466,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-19.us-east-2.compute.internal,}"
Oct 30 19:05:45.477817 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.477767    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwf8n"
Oct 30 19:05:45.483974 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.483944    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 19:05:45.483974 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.483954    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-19.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:05:45.484138 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.484104    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:05:45.484626 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.484610    2578 manager.go:324] Recovery completed
Oct 30 19:05:45.486865 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.486846    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 19:05:45.492623 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.492607    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:45.494559 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.494540    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:45.494665 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.494594    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:45.494665 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.494609    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:45.495211 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.495197    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:05:45.495211 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.495210    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:05:45.495280 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.495225    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:05:45.495314 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.495199    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:45.496049 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.496029    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:45.496106 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.496057    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:45.496106 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.496071    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:45.496290 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.496280    2578 policy_none.go:50] "Start"
Oct 30 19:05:45.496322 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.496294    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:05:45.496322 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.496304    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:05:45.497286 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.497277    2578 policy_none.go:44] "Start"
Oct 30 19:05:45.535255 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.535222    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:05:45.537695 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.537676    2578 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:05:45.537809 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.537714    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:05:45.537809 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.537727    2578 server.go:87] "Starting device plugin registration server"
Oct 30 19:05:45.538029 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.538015    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:05:45.538125 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.538028    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:05:45.538186 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.538143    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:05:45.538240 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.538224    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:05:45.538240 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.538232    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:05:45.539196 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.539178    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:05:45.539260 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.539229    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:45.652854 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.638772    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:45.652854 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.643082    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:45.652854 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.643115    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:45.652854 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.643131    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:45.652854 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.643168    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.652854 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.651066    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.652854 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.651092    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-19.us-east-2.compute.internal\": node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:45.669342 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.669318    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:45.716400 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.716354    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:05:45.717489 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.717468    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:05:45.717543 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.717499    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:05:45.717543 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.717527    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:05:45.718154 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.717568    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:05:45.733905 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.733878    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:45.770021 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.769993    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:45.818255 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.818212    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"]
Oct 30 19:05:45.818345 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.818293    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:45.819372 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.819353    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:45.819480 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.819387    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:45.819480 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.819400    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:45.820621 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.820608    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:45.820793 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.820777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.820854 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.820811    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:45.821413 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.821390    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:45.821502 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.821422    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:45.821502 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.821426    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:45.821502 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.821441    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:45.821502 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.821453    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:45.821502 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.821441    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:45.822647 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.822623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.822701 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.822663    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:45.823421 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.823405    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:45.823512 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.823432    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:45.823512 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.823446    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:45.853706 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.853680    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found" node="ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.857399 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.857379    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found" node="ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.870416 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.870379    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:45.871518 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.871497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.871589 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.871526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.871589 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.871553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b20bfab463977c229c9db20c57c04c0b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"b20bfab463977c229c9db20c57c04c0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.971426 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:45.971346    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:45.972481 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.972462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.972530 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.972490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.972530 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.972508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b20bfab463977c229c9db20c57c04c0b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"b20bfab463977c229c9db20c57c04c0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.972595 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.972538    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b20bfab463977c229c9db20c57c04c0b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"b20bfab463977c229c9db20c57c04c0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.972595 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.972552    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:45.972595 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:45.972574    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:46.071750 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:46.071707    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:46.157027 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:46.157005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:46.160907 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:46.160572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:46.172348 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:46.172183    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:46.272704 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:46.272611    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:46.337856 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:46.337829    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:46.373223 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:46.373191    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:47.726893 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:47.726869    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:05:47.727388 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:47.727075    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:05:47.727388 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:47.727085    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:05:47.794554 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:47.794520    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:47.803069 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:47.803004    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:00:46 +0000 UTC" deadline="2028-04-04 16:45:32.62031248 +0000 UTC"
Oct 30 19:05:47.803069 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:47.803050    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12525h39m44.817266s"
Oct 30 19:05:47.895046 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:47.894945    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:47.995356 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:47.995324    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:48.054451 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:48.054413    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617112a22c78eeb8b0062c43970103d2.slice/crio-8389b33f6f123f5ffab77d0a7ac3f81ba9b7460ebc6892248ab3bdb028ae42eb WatchSource:0}: Error finding container 8389b33f6f123f5ffab77d0a7ac3f81ba9b7460ebc6892248ab3bdb028ae42eb: Status 404 returned error can't find the container with id 8389b33f6f123f5ffab77d0a7ac3f81ba9b7460ebc6892248ab3bdb028ae42eb
Oct 30 19:05:48.055002 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:48.054967    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20bfab463977c229c9db20c57c04c0b.slice/crio-4764481df7a51d46ba985f6d991c7cb4ba0bc7a616154ce8f03af416e9b3d0ba WatchSource:0}: Error finding container 4764481df7a51d46ba985f6d991c7cb4ba0bc7a616154ce8f03af416e9b3d0ba: Status 404 returned error can't find the container with id 4764481df7a51d46ba985f6d991c7cb4ba0bc7a616154ce8f03af416e9b3d0ba
Oct 30 19:05:48.060305 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.060218    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:05:48.095973 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:48.095944    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:48.105458 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.105439    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:48.196513 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:48.196434    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:05:48.278462 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.278435    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:48.290505 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.290479    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:48.303014 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.302974    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:05:48.303998 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.303984    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:05:48.313367 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.313350    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:05:48.766373 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.766341    2578 apiserver.go:52] "Watching apiserver"
Oct 30 19:05:48.777390 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.777358    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:05:48.777748 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.777722    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-5bdll","openshift-image-registry/node-ca-9b5ds","openshift-network-diagnostics/network-check-target-k4plb","kube-system/konnectivity-agent-24ljs","kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb","openshift-multus/multus-v9ps7","openshift-multus/network-metrics-daemon-tbsh8","kube-system/global-pull-secret-syncer-tvwkt","openshift-cluster-node-tuning-operator/tuned-ml5wz","openshift-network-operator/iptables-alerter-ctfbf","openshift-ovn-kubernetes/ovnkube-node-ldfm8","openshift-dns/node-resolver-829qt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"]
Oct 30 19:05:48.779420 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.779391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.781082 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.781040    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.782069 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.782032    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.783280 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.783257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.784448 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.784334    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:05:48.784448 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.784394    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:05:48.784448 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.784394    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:05:48.784448 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.784449    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:05:48.784696 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.784562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:48.784696 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.784599    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gkzrx\""
Oct 30 19:05:48.784696 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.784687    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:05:48.784909 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.784889    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hsp67\""
Oct 30 19:05:48.784994 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.784962    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:05:48.785107 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.785090    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:05:48.785185 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.785140    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ngr9b\""
Oct 30 19:05:48.785693 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.785674    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.786394 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.786373    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:05:48.786492 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.786436    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xd9sf\""
Oct 30 19:05:48.786492 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.786447    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:05:48.786605 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.786541    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:05:48.786732 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.786714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.787613 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.787598    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4z6gp\""
Oct 30 19:05:48.787860 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.787845    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:05:48.787974 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.787894    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:05:48.788427 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.788117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:48.788427 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.788183    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w2h58\""
Oct 30 19:05:48.788427 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:48.788178    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:05:48.788739 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.788720    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:05:48.788739 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.788730    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:05:48.788849 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.788785    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:05:48.789683 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.789663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:48.789772 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:48.789723    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:05:48.790297 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.790278    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-m74bb\""
Oct 30 19:05:48.790376 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.790342    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:05:48.791669 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.791172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.791908 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.791886    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:05:48.791908 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.791907    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:05:48.792078 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.791965    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:05:48.792078 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.791984    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:05:48.792645 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.792625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.793838 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.793817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:48.793921 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:48.793880    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:05:48.799892 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.799871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.800003 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.799907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:48.800003 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.799974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-host\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.800144 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-netns\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.800144 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ac8869-277b-44e4-a31a-bde13751efd9-tmp-dir\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.800249 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-multus\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.800249 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52v6\" (UniqueName: \"kubernetes.io/projected/38ac8869-277b-44e4-a31a-bde13751efd9-kube-api-access-g52v6\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.800249 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b9c7068-7163-4c61-8a7f-5ec789247fa7-host\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.800369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-cni-binary-copy\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.800369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-k8s-cni-cncf-io\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.800369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-multus-certs\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.800369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-socket-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.800504 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6wg\" (UniqueName: \"kubernetes.io/projected/0b9c7068-7163-4c61-8a7f-5ec789247fa7-kube-api-access-dw6wg\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.800504 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-kubelet\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.800504 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysconfig\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.800504 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-agent-certs\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:48.800638 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-bin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.800638 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46dd1c02-246a-41f2-bb6a-7523cc8b0436-iptables-alerter-script\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.800638 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:48.800742 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmwz8\" (UniqueName: \"kubernetes.io/projected/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-kube-api-access-bmwz8\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:48.800742 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.800742 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-tmp\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.800863 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbk85\" (UniqueName: \"kubernetes.io/projected/5232cfc3-47d5-46bb-8181-ecd05bc02348-kube-api-access-mbk85\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.800863 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-kubelet-config\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:48.800863 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.800985 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-run\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.800985 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.800940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-socket-dir-parent\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.801101 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-conf-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.801101 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-dbus\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:48.801101 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-sys-fs\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.801247 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cnibin\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.801247 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-lib-modules\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.801247 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-var-lib-kubelet\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.801247 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-tuned\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.801247 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqhq\" (UniqueName: \"kubernetes.io/projected/46dd1c02-246a-41f2-bb6a-7523cc8b0436-kube-api-access-csqhq\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.801247 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-konnectivity-ca\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:48.801247 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-hostroot\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.801530 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.801251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-modprobe-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-kubernetes\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-system-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38ac8869-277b-44e4-a31a-bde13751efd9-hosts-file\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-system-cni-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-conf\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-etc-kubernetes\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-registration-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgsx5\" (UniqueName: \"kubernetes.io/projected/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-kube-api-access-mgsx5\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.803999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b9c7068-7163-4c61-8a7f-5ec789247fa7-serviceca\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-cnibin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-daemon-config\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp66z\" (UniqueName: \"kubernetes.io/projected/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kube-api-access-tp66z\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.805822 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.806799 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-systemd\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.806799 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-sys\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.806799 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dd1c02-246a-41f2-bb6a-7523cc8b0436-host-slash\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.806799 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.806799 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tmhg\" (UniqueName: \"kubernetes.io/projected/bf9f6e91-3cab-4b47-9526-a50b62527a16-kube-api-access-9tmhg\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.806799 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.804515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-device-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.807356 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.807306    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:05:48.824043 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.824014    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vww6l"
Oct 30 19:05:48.836930 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.836909    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vww6l"
Oct 30 19:05:48.884965 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.884932    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:05:48.885170 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.885044    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:05:48.885170 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.885070    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:05:48.885170 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.885109    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:05:48.885170 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.885136    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:05:48.885377 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.885357    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:05:48.885754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.885680    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:05:48.885754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.885723    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jks97\""
Oct 30 19:05:48.885898 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.885763    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:05:48.885898 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.885771    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xlxzb\""
Oct 30 19:05:48.891998 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.891931    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:05:48.905318 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.905477 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-run\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.905477 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-socket-dir-parent\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.905477 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-conf-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.905477 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-dbus\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:48.905477 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-sys-fs\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.905477 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cnibin\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-lib-modules\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-var-lib-kubelet\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-tuned\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqhq\" (UniqueName: \"kubernetes.io/projected/46dd1c02-246a-41f2-bb6a-7523cc8b0436-kube-api-access-csqhq\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-konnectivity-ca\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-hostroot\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-netns\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-modprobe-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-kubernetes\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-system-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38ac8869-277b-44e4-a31a-bde13751efd9-hosts-file\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.905721 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-systemd-units\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-bin\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-netd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-env-overrides\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-system-cni-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-conf\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-etc-kubernetes\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-registration-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905957    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.905953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-node-log\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsx5\" (UniqueName: \"kubernetes.io/projected/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-kube-api-access-mgsx5\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.906203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906155    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-socket-dir-parent\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906159    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-run\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b9c7068-7163-4c61-8a7f-5ec789247fa7-serviceca\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906184    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-conf-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-cnibin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906291    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-lib-modules\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906328    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-dbus\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-daemon-config\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38ac8869-277b-44e4-a31a-bde13751efd9-hosts-file\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906397    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-system-cni-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp66z\" (UniqueName: \"kubernetes.io/projected/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kube-api-access-tp66z\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906433    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cnibin\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906464    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-kubernetes\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906476    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-sys-fs\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-slash\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906503    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-systemd\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.906816 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-sys\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dd1c02-246a-41f2-bb6a-7523cc8b0436-host-slash\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906606    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-conf\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tmhg\" (UniqueName: \"kubernetes.io/projected/bf9f6e91-3cab-4b47-9526-a50b62527a16-kube-api-access-9tmhg\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-device-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906708    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-device-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906739    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-system-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906774    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-sys\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906808    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dd1c02-246a-41f2-bb6a-7523cc8b0436-host-slash\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906905    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.906964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907006    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-registration-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907017    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-hostroot\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907025    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-systemd\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-host\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.907678 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907150    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-modprobe-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907148    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-netns\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907193    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-etc-kubernetes\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907210    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ac8869-277b-44e4-a31a-bde13751efd9-tmp-dir\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907277    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-netns\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-etc-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovn-node-metrics-cert\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-multus\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52v6\" (UniqueName: \"kubernetes.io/projected/38ac8869-277b-44e4-a31a-bde13751efd9-kube-api-access-g52v6\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907452    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-daemon-config\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907470    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-konnectivity-ca\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b9c7068-7163-4c61-8a7f-5ec789247fa7-host\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907540    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.908475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907553    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-cni-binary-copy\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-k8s-cni-cncf-io\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907607    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-var-lib-kubelet\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907585    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-cnibin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7l5r\" (UniqueName: \"kubernetes.io/projected/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-kube-api-access-z7l5r\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907594    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-host\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907642    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ac8869-277b-44e4-a31a-bde13751efd9-tmp-dir\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:48.907658    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907674    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-k8s-cni-cncf-io\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907675    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b9c7068-7163-4c61-8a7f-5ec789247fa7-serviceca\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907693    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-multus\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:48.907731    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs podName:ceeebf2b-9b37-4988-b6c1-b8b41e09449a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:49.407697528 +0000 UTC m=+2.971160915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs") pod "network-metrics-daemon-tbsh8" (UID: "ceeebf2b-9b37-4988-b6c1-b8b41e09449a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-multus-certs\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907812    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b9c7068-7163-4c61-8a7f-5ec789247fa7-host\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907829    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-multus-certs\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-socket-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.909207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6wg\" (UniqueName: \"kubernetes.io/projected/0b9c7068-7163-4c61-8a7f-5ec789247fa7-kube-api-access-dw6wg\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.907952    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-socket-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-kubelet\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-kubelet\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908102    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-kubelet\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-cni-binary-copy\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-systemd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-script-lib\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysconfig\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-agent-certs\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-bin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908385    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysconfig\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-ovn\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908494    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-bin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46dd1c02-246a-41f2-bb6a-7523cc8b0436-iptables-alerter-script\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmwz8\" (UniqueName: \"kubernetes.io/projected/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-kube-api-access-bmwz8\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:48.909867 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-log-socket\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-tmp\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbk85\" (UniqueName: \"kubernetes.io/projected/5232cfc3-47d5-46bb-8181-ecd05bc02348-kube-api-access-mbk85\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-kubelet-config\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-var-lib-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:48.908963    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.908969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-config\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:48.909024    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret podName:baec05dd-6d69-41d9-ad73-4179b17ba8f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:49.409008385 +0000 UTC m=+2.972471742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret") pod "global-pull-secret-syncer-tvwkt" (UID: "baec05dd-6d69-41d9-ad73-4179b17ba8f4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.909097    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.909244    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-kubelet-config\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:48.910472 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.909428    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46dd1c02-246a-41f2-bb6a-7523cc8b0436-iptables-alerter-script\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.911763 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.911741    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-tuned\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.911855 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.911770    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-tmp\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:48.912260 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.912241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-agent-certs\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:48.918750 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.918726    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsx5\" (UniqueName: \"kubernetes.io/projected/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-kube-api-access-mgsx5\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:48.919757 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.919731    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tmhg\" (UniqueName: \"kubernetes.io/projected/bf9f6e91-3cab-4b47-9526-a50b62527a16-kube-api-access-9tmhg\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:48.921274 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.921253    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp66z\" (UniqueName: \"kubernetes.io/projected/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kube-api-access-tp66z\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:48.922204 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.922186    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqhq\" (UniqueName: \"kubernetes.io/projected/46dd1c02-246a-41f2-bb6a-7523cc8b0436-kube-api-access-csqhq\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:48.926698 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.926674    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52v6\" (UniqueName: \"kubernetes.io/projected/38ac8869-277b-44e4-a31a-bde13751efd9-kube-api-access-g52v6\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:48.927697 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.927677    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmwz8\" (UniqueName: \"kubernetes.io/projected/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-kube-api-access-bmwz8\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:48.927697 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.927686    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6wg\" (UniqueName: \"kubernetes.io/projected/0b9c7068-7163-4c61-8a7f-5ec789247fa7-kube-api-access-dw6wg\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:48.933293 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:48.929543    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbk85\" (UniqueName: \"kubernetes.io/projected/5232cfc3-47d5-46bb-8181-ecd05bc02348-kube-api-access-mbk85\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:49.010022 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.009979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-etc-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010022 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovn-node-metrics-cert\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010087    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-etc-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l5r\" (UniqueName: \"kubernetes.io/projected/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-kube-api-access-z7l5r\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-kubelet\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-systemd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-script-lib\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-ovn\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010201    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010235 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-log-socket\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010274    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-kubelet\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010304    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-systemd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010477    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-ovn\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-var-lib-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-config\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-netns\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-systemd-units\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-bin\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-netd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-env-overrides\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-node-log\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-slash\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010712    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-slash\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.010754 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010733    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-var-lib-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011125 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.010850    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-script-lib\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011125 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011081    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-node-log\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011125 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011093    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-log-socket\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011125 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011107    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011125 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011294 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011239    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-netd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011333 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011289    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-netns\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011333 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011325    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-systemd-units\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011410 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011330    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-bin\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011640 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011621    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-config\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.011747 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.011728    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-env-overrides\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.013039 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.013018    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovn-node-metrics-cert\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.019895 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.019830    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:49.019895 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.019859    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:49.019895 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.019871    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b5px2 for pod openshift-network-diagnostics/network-check-target-k4plb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:49.020132 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.019945    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2 podName:6e38a783-1270-4703-a97c-cabdbe1c50ce nodeName:}" failed. No retries permitted until 2026-10-30 19:05:49.519926458 +0000 UTC m=+3.083389830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b5px2" (UniqueName: "kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2") pod "network-check-target-k4plb" (UID: "6e38a783-1270-4703-a97c-cabdbe1c50ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:49.022454 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.022429    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7l5r\" (UniqueName: \"kubernetes.io/projected/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-kube-api-access-z7l5r\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.042770 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.042712    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"617112a22c78eeb8b0062c43970103d2","Type":"ContainerStarted","Data":"8389b33f6f123f5ffab77d0a7ac3f81ba9b7460ebc6892248ab3bdb028ae42eb"}
Oct 30 19:05:49.043838 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.043808    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"b20bfab463977c229c9db20c57c04c0b","Type":"ContainerStarted","Data":"4764481df7a51d46ba985f6d991c7cb4ba0bc7a616154ce8f03af416e9b3d0ba"}
Oct 30 19:05:49.093807 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.093770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:05:49.102778 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:49.102741    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5232cfc3_47d5_46bb_8181_ecd05bc02348.slice/crio-3782872d278a9d6901dbba9610bbbcfb4c5df5421b744df3361395acb095dbe2 WatchSource:0}: Error finding container 3782872d278a9d6901dbba9610bbbcfb4c5df5421b744df3361395acb095dbe2: Status 404 returned error can't find the container with id 3782872d278a9d6901dbba9610bbbcfb4c5df5421b744df3361395acb095dbe2
Oct 30 19:05:49.107159 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.107132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:49.111049 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.111021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:05:49.116238 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:49.116180    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b26e19e_b8e9_496a_bfcb_6cd5add61e6e.slice/crio-57efde00068fa6c7bda32f3d95f526a5deee058013fe9d9755b61b1ad197587b WatchSource:0}: Error finding container 57efde00068fa6c7bda32f3d95f526a5deee058013fe9d9755b61b1ad197587b: Status 404 returned error can't find the container with id 57efde00068fa6c7bda32f3d95f526a5deee058013fe9d9755b61b1ad197587b
Oct 30 19:05:49.117786 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.117763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-829qt"
Oct 30 19:05:49.124930 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:49.120865    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9c7068_7163_4c61_8a7f_5ec789247fa7.slice/crio-91219e0e05ef48c11ad60ccf1335cbe7b1f267559355d90e6a84f839a4017459 WatchSource:0}: Error finding container 91219e0e05ef48c11ad60ccf1335cbe7b1f267559355d90e6a84f839a4017459: Status 404 returned error can't find the container with id 91219e0e05ef48c11ad60ccf1335cbe7b1f267559355d90e6a84f839a4017459
Oct 30 19:05:49.127397 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.126588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:05:49.127397 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:49.126899    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ac8869_277b_44e4_a31a_bde13751efd9.slice/crio-87dbbf18cd6875c46343477a1d5e975f5a46906ad5e7219ba14af2f603206796 WatchSource:0}: Error finding container 87dbbf18cd6875c46343477a1d5e975f5a46906ad5e7219ba14af2f603206796: Status 404 returned error can't find the container with id 87dbbf18cd6875c46343477a1d5e975f5a46906ad5e7219ba14af2f603206796
Oct 30 19:05:49.136282 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.136258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:05:49.140775 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.140590    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v9ps7"
Oct 30 19:05:49.146244 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:49.146212    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb1844b_cbea_4ef6_a608_340b7ae7fee5.slice/crio-374a49a14c9b9712e1031be891f6677e3e5455bae39de6b4611ad49abca16d76 WatchSource:0}: Error finding container 374a49a14c9b9712e1031be891f6677e3e5455bae39de6b4611ad49abca16d76: Status 404 returned error can't find the container with id 374a49a14c9b9712e1031be891f6677e3e5455bae39de6b4611ad49abca16d76
Oct 30 19:05:49.152209 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:49.152164    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9f6e91_3cab_4b47_9526_a50b62527a16.slice/crio-07a73e09991a17edb14493abc70a24254ae29627a3b75ddbf6c0140d526aab4a WatchSource:0}: Error finding container 07a73e09991a17edb14493abc70a24254ae29627a3b75ddbf6c0140d526aab4a: Status 404 returned error can't find the container with id 07a73e09991a17edb14493abc70a24254ae29627a3b75ddbf6c0140d526aab4a
Oct 30 19:05:49.160087 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.160047    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:05:49.166848 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.166800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:05:49.169735 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:49.169707    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6585f59_5f16_4766_a3a2_e8aeac2c4bcb.slice/crio-be123be39dc836e731c1d92f23e439dfa8e616a02caae81dc7ae272b875c0e94 WatchSource:0}: Error finding container be123be39dc836e731c1d92f23e439dfa8e616a02caae81dc7ae272b875c0e94: Status 404 returned error can't find the container with id be123be39dc836e731c1d92f23e439dfa8e616a02caae81dc7ae272b875c0e94
Oct 30 19:05:49.177329 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:05:49.177304    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31e88aa_14a5_4deb_a503_17b7f52cb4b9.slice/crio-d2525ac23364b8fad679e6fe5b360b73de2db27308552e63d46d72da7a3236fa WatchSource:0}: Error finding container d2525ac23364b8fad679e6fe5b360b73de2db27308552e63d46d72da7a3236fa: Status 404 returned error can't find the container with id d2525ac23364b8fad679e6fe5b360b73de2db27308552e63d46d72da7a3236fa
Oct 30 19:05:49.225173 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.225142    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:49.414147 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.414052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:49.414147 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.414136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:49.414359 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.414183    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:49.414359 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.414230    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:49.414359 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.414267    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs podName:ceeebf2b-9b37-4988-b6c1-b8b41e09449a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:50.414247377 +0000 UTC m=+3.977710752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs") pod "network-metrics-daemon-tbsh8" (UID: "ceeebf2b-9b37-4988-b6c1-b8b41e09449a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:49.414359 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.414290    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret podName:baec05dd-6d69-41d9-ad73-4179b17ba8f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:50.414277588 +0000 UTC m=+3.977740949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret") pod "global-pull-secret-syncer-tvwkt" (UID: "baec05dd-6d69-41d9-ad73-4179b17ba8f4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:49.615768 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.615148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:49.615768 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.615294    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:49.615768 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.615319    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:49.615768 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.615332    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b5px2 for pod openshift-network-diagnostics/network-check-target-k4plb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:49.615768 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:49.615406    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2 podName:6e38a783-1270-4703-a97c-cabdbe1c50ce nodeName:}" failed. No retries permitted until 2026-10-30 19:05:50.615378786 +0000 UTC m=+4.178842162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5px2" (UniqueName: "kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2") pod "network-check-target-k4plb" (UID: "6e38a783-1270-4703-a97c-cabdbe1c50ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:49.839841 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.839745    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:00:48 +0000 UTC" deadline="2028-07-26 08:38:06.613448814 +0000 UTC"
Oct 30 19:05:49.839841 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:49.839781    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15229h32m16.773672375s"
Oct 30 19:05:50.039016 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.038897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:50.039203 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:50.039083    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:05:50.070976 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.070819    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-829qt" event={"ID":"38ac8869-277b-44e4-a31a-bde13751efd9","Type":"ContainerStarted","Data":"87dbbf18cd6875c46343477a1d5e975f5a46906ad5e7219ba14af2f603206796"}
Oct 30 19:05:50.083450 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.083343    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ml5wz" event={"ID":"5232cfc3-47d5-46bb-8181-ecd05bc02348","Type":"ContainerStarted","Data":"3782872d278a9d6901dbba9610bbbcfb4c5df5421b744df3361395acb095dbe2"}
Oct 30 19:05:50.092006 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.091965    2578 generic.go:356] "Generic (PLEG): container finished" podID="617112a22c78eeb8b0062c43970103d2" containerID="64405d5dff437f04cf8c54a0467494cc4cd3437fff3df0018c9ec3af1d6b97dd" exitCode=0
Oct 30 19:05:50.092177 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.092081    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"617112a22c78eeb8b0062c43970103d2","Type":"ContainerDied","Data":"64405d5dff437f04cf8c54a0467494cc4cd3437fff3df0018c9ec3af1d6b97dd"}
Oct 30 19:05:50.105792 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.105666    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" event={"ID":"f31e88aa-14a5-4deb-a503-17b7f52cb4b9","Type":"ContainerStarted","Data":"d2525ac23364b8fad679e6fe5b360b73de2db27308552e63d46d72da7a3236fa"}
Oct 30 19:05:50.115896 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.115817    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v9ps7" event={"ID":"bf9f6e91-3cab-4b47-9526-a50b62527a16","Type":"ContainerStarted","Data":"07a73e09991a17edb14493abc70a24254ae29627a3b75ddbf6c0140d526aab4a"}
Oct 30 19:05:50.123907 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.123819    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb" event={"ID":"cbb1844b-cbea-4ef6-a608-340b7ae7fee5","Type":"ContainerStarted","Data":"374a49a14c9b9712e1031be891f6677e3e5455bae39de6b4611ad49abca16d76"}
Oct 30 19:05:50.128752 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.128716    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctfbf" event={"ID":"46dd1c02-246a-41f2-bb6a-7523cc8b0436","Type":"ContainerStarted","Data":"51f82c3dcd4f9f885f327fdfdd2cc76652dbe882b89beded02217427b1d72be6"}
Oct 30 19:05:50.131468 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.131436    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9b5ds" event={"ID":"0b9c7068-7163-4c61-8a7f-5ec789247fa7","Type":"ContainerStarted","Data":"91219e0e05ef48c11ad60ccf1335cbe7b1f267559355d90e6a84f839a4017459"}
Oct 30 19:05:50.139009 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.138959    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24ljs" event={"ID":"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e","Type":"ContainerStarted","Data":"57efde00068fa6c7bda32f3d95f526a5deee058013fe9d9755b61b1ad197587b"}
Oct 30 19:05:50.144021 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.143331    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"b20bfab463977c229c9db20c57c04c0b","Type":"ContainerStarted","Data":"cba1b9dd034e4082bb1912ee3b0ceb0033c2559f23eea6cf75113ff342d3db38"}
Oct 30 19:05:50.159795 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.159679    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal" podStartSLOduration=2.15965935 podStartE2EDuration="2.15965935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:50.159132723 +0000 UTC m=+3.722596103" watchObservedRunningTime="2026-10-30 19:05:50.15965935 +0000 UTC m=+3.723122732"
Oct 30 19:05:50.160779 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.160593    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdll" event={"ID":"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb","Type":"ContainerStarted","Data":"be123be39dc836e731c1d92f23e439dfa8e616a02caae81dc7ae272b875c0e94"}
Oct 30 19:05:50.421073 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.421012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:50.421247 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.421135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:50.421302 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:50.421290    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:50.421369 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:50.421355    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs podName:ceeebf2b-9b37-4988-b6c1-b8b41e09449a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:52.421334602 +0000 UTC m=+5.984797981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs") pod "network-metrics-daemon-tbsh8" (UID: "ceeebf2b-9b37-4988-b6c1-b8b41e09449a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:50.422689 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:50.422666    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:50.422793 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:50.422734    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret podName:baec05dd-6d69-41d9-ad73-4179b17ba8f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:52.422717389 +0000 UTC m=+5.986180754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret") pod "global-pull-secret-syncer-tvwkt" (UID: "baec05dd-6d69-41d9-ad73-4179b17ba8f4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:50.623539 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.623417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:50.623711 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:50.623568    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:50.623711 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:50.623583    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:50.623711 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:50.623592    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b5px2 for pod openshift-network-diagnostics/network-check-target-k4plb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:50.623711 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:50.623635    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2 podName:6e38a783-1270-4703-a97c-cabdbe1c50ce nodeName:}" failed. No retries permitted until 2026-10-30 19:05:52.623623558 +0000 UTC m=+6.187086916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5px2" (UniqueName: "kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2") pod "network-check-target-k4plb" (UID: "6e38a783-1270-4703-a97c-cabdbe1c50ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:50.835900 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.835867    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:50.840244 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.840154    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:00:48 +0000 UTC" deadline="2028-06-15 11:55:58.280353636 +0000 UTC"
Oct 30 19:05:50.840244 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:50.840244    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14248h50m7.440113817s"
Oct 30 19:05:51.039119 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:51.038736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:51.039119 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:51.038878    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:05:51.039334 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:51.039170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:51.039334 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:51.039273    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:05:51.174360 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:51.174324    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"617112a22c78eeb8b0062c43970103d2","Type":"ContainerStarted","Data":"f07cecb5226ed8c7e0b7aed9c278386f2cf37f1475be14f7ef1965c679c0f5b2"}
Oct 30 19:05:52.039121 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:52.038989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:52.039580 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:52.039149    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:05:52.181904 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:52.181792    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdll" event={"ID":"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb","Type":"ContainerStarted","Data":"9e024d2561088966412594a34df6b9eb77f51f3233e545f2172ca1cb473336e7"}
Oct 30 19:05:52.206268 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:52.205524    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal" podStartSLOduration=4.205502798 podStartE2EDuration="4.205502798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:51.191308972 +0000 UTC m=+4.754772353" watchObservedRunningTime="2026-10-30 19:05:52.205502798 +0000 UTC m=+5.768966182"
Oct 30 19:05:52.438366 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:52.438277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:52.438366 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:52.438359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:52.438606 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:52.438511    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:52.438606 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:52.438576    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs podName:ceeebf2b-9b37-4988-b6c1-b8b41e09449a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:56.438558396 +0000 UTC m=+10.002021769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs") pod "network-metrics-daemon-tbsh8" (UID: "ceeebf2b-9b37-4988-b6c1-b8b41e09449a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:52.439089 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:52.439071    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:52.439198 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:52.439127    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret podName:baec05dd-6d69-41d9-ad73-4179b17ba8f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:56.439111178 +0000 UTC m=+10.002574550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret") pod "global-pull-secret-syncer-tvwkt" (UID: "baec05dd-6d69-41d9-ad73-4179b17ba8f4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:52.640994 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:52.640364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:52.640994 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:52.640563    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:52.640994 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:52.640586    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:52.640994 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:52.640598    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b5px2 for pod openshift-network-diagnostics/network-check-target-k4plb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:52.640994 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:52.640656    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2 podName:6e38a783-1270-4703-a97c-cabdbe1c50ce nodeName:}" failed. No retries permitted until 2026-10-30 19:05:56.640636479 +0000 UTC m=+10.204099848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5px2" (UniqueName: "kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2") pod "network-check-target-k4plb" (UID: "6e38a783-1270-4703-a97c-cabdbe1c50ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:53.041502 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:53.041097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:53.041502 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:53.041219    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:05:53.041502 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:53.041275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:53.041502 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:53.041338    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:05:53.186233 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:53.186202    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="9e024d2561088966412594a34df6b9eb77f51f3233e545f2172ca1cb473336e7" exitCode=0
Oct 30 19:05:53.186333 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:53.186248    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdll" event={"ID":"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb","Type":"ContainerDied","Data":"9e024d2561088966412594a34df6b9eb77f51f3233e545f2172ca1cb473336e7"}
Oct 30 19:05:54.039082 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:54.038509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:54.039082 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:54.038652    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:05:54.194601 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:54.194512    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9b5ds" event={"ID":"0b9c7068-7163-4c61-8a7f-5ec789247fa7","Type":"ContainerStarted","Data":"91d11ff32c76571976c55b0f18f8d091652be89a73de7ac04ba73312ed718212"}
Oct 30 19:05:54.201439 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:54.200905    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24ljs" event={"ID":"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e","Type":"ContainerStarted","Data":"5e4b3430412d574b5a97881e25c7f0c012fc2f14b0f503afb5209120d177b027"}
Oct 30 19:05:54.228442 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:54.227027    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9b5ds" podStartSLOduration=3.532006587 podStartE2EDuration="8.226988235s" totalImagesPullingTime="4.694981648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:54.211568283 +0000 UTC m=+7.775031642" watchObservedRunningTime="2026-10-30 19:05:54.226988235 +0000 UTC m=+7.790451615"
Oct 30 19:05:55.040790 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:55.040758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:55.040964 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:55.040931    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:05:55.041379 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:55.041358    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:55.041633 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:55.041481    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:05:55.206026 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:55.205984    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb" event={"ID":"cbb1844b-cbea-4ef6-a608-340b7ae7fee5","Type":"ContainerStarted","Data":"465da531a44525e4d2159e35c45d83298375cc858d0aa973088b047024128fa5"}
Oct 30 19:05:56.038907 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:56.038387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:56.038907 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:56.038537    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:05:56.478468 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:56.478431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:56.478896 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:56.478505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:56.478896 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:56.478633    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:56.478896 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:56.478693    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret podName:baec05dd-6d69-41d9-ad73-4179b17ba8f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:04.478675575 +0000 UTC m=+18.042138948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret") pod "global-pull-secret-syncer-tvwkt" (UID: "baec05dd-6d69-41d9-ad73-4179b17ba8f4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:56.479188 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:56.479168    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:56.479251 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:56.479229    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs podName:ceeebf2b-9b37-4988-b6c1-b8b41e09449a nodeName:}" failed. No retries permitted until 2026-10-30 19:06:04.479213027 +0000 UTC m=+18.042676406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs") pod "network-metrics-daemon-tbsh8" (UID: "ceeebf2b-9b37-4988-b6c1-b8b41e09449a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:56.560257 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:56.559492    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:05:56.679595 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:56.679506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:56.679765 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:56.679658    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:56.679765 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:56.679678    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:56.679765 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:56.679690    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b5px2 for pod openshift-network-diagnostics/network-check-target-k4plb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:56.679765 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:56.679755    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2 podName:6e38a783-1270-4703-a97c-cabdbe1c50ce nodeName:}" failed. No retries permitted until 2026-10-30 19:06:04.679737263 +0000 UTC m=+18.243200637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5px2" (UniqueName: "kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2") pod "network-check-target-k4plb" (UID: "6e38a783-1270-4703-a97c-cabdbe1c50ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:56.867549 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:56.867426    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:05:56.559518575Z","UUID":"492d0b1a-460f-4bcb-b447-3fcf38a6b588","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:05:56.869784 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:56.869298    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:05:56.869784 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:56.869330    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:05:57.040188 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:57.040146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:57.040394 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:57.040368    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:05:57.040451 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:57.040397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:57.040499 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:57.040464    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:05:57.214859 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:57.214822    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v9ps7" event={"ID":"bf9f6e91-3cab-4b47-9526-a50b62527a16","Type":"ContainerStarted","Data":"68d4da8d81977d416d85ad54d8b51a1026b10d9eca6bb47958dd225441e3b945"}
Oct 30 19:05:57.219155 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:57.219122    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb" event={"ID":"cbb1844b-cbea-4ef6-a608-340b7ae7fee5","Type":"ContainerStarted","Data":"2037d24eefcca06197a641711e7de8351930507691acbdeed2eb632a21ae4a75"}
Oct 30 19:05:57.240376 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:57.240326    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-24ljs" podStartSLOduration=6.336254119 podStartE2EDuration="10.24030919s" totalImagesPullingTime="3.904055071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:54.228400871 +0000 UTC m=+7.791864280" watchObservedRunningTime="2026-10-30 19:05:57.24030919 +0000 UTC m=+10.803772569"
Oct 30 19:05:57.240536 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:57.240489    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-v9ps7" podStartSLOduration=2.892704643 podStartE2EDuration="10.240483621s" totalImagesPullingTime="7.347778978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:57.240084961 +0000 UTC m=+10.803548354" watchObservedRunningTime="2026-10-30 19:05:57.240483621 +0000 UTC m=+10.803947065"
Oct 30 19:05:58.039098 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:58.039040    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:05:58.039540 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:58.039206    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:05:58.226325 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:58.226288    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb" event={"ID":"cbb1844b-cbea-4ef6-a608-340b7ae7fee5","Type":"ContainerStarted","Data":"012d9c9b0b5d86bf8050bad2028c4eb7d52cadc2089c1012533b15313eaa7ae4"}
Oct 30 19:05:58.228823 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:58.228794    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="16e6103eabf6954a78bba6ca09877442b8eaddd5828fe85d949f243a929490be" exitCode=0
Oct 30 19:05:58.228988 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:58.228880    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdll" event={"ID":"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb","Type":"ContainerDied","Data":"16e6103eabf6954a78bba6ca09877442b8eaddd5828fe85d949f243a929490be"}
Oct 30 19:05:58.230843 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:58.230784    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-829qt" event={"ID":"38ac8869-277b-44e4-a31a-bde13751efd9","Type":"ContainerStarted","Data":"3588cf8e60094de0e71e15d6804cccedaf94d210dbc1231400815888b68ad262"}
Oct 30 19:05:58.247296 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:58.247252    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-829qt" podStartSLOduration=4.333952109 podStartE2EDuration="12.247238368s" totalImagesPullingTime="7.913286259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:57.272716213 +0000 UTC m=+10.836179592" watchObservedRunningTime="2026-10-30 19:05:58.247238368 +0000 UTC m=+11.810701744"
Oct 30 19:05:58.247670 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:58.247638    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb" podStartSLOduration=3.194938023 podStartE2EDuration="11.24762893s" totalImagesPullingTime="8.052690907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:58.247079271 +0000 UTC m=+11.810542655" watchObservedRunningTime="2026-10-30 19:05:58.24762893 +0000 UTC m=+11.811092309"
Oct 30 19:05:58.565431 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:58.565389    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:58.566072 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:58.566039    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:59.039004 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:59.038746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:05:59.039176 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:59.038804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:05:59.039176 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:59.039135    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:05:59.039579 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:05:59.039197    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:05:59.234964 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:59.234868    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ml5wz" event={"ID":"5232cfc3-47d5-46bb-8181-ecd05bc02348","Type":"ContainerStarted","Data":"f8372614c37f3af30d71119a0718a1415af9198aa5019e2e531bf3da053cb953"}
Oct 30 19:05:59.236686 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:59.236660    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctfbf" event={"ID":"46dd1c02-246a-41f2-bb6a-7523cc8b0436","Type":"ContainerStarted","Data":"fe46fbfa2cdf41af20a09b458e4cb90d22bbb4dae0ea263565e84228b44866c6"}
Oct 30 19:05:59.237233 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:59.237210    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:59.237744 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:59.237724    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:05:59.253942 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:59.253882    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ml5wz" podStartSLOduration=3.549006507 podStartE2EDuration="13.253867502s" totalImagesPullingTime="9.704860995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:59.253372181 +0000 UTC m=+12.816835560" watchObservedRunningTime="2026-10-30 19:05:59.253867502 +0000 UTC m=+12.817330880"
Oct 30 19:05:59.287738 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:05:59.287662    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ctfbf" podStartSLOduration=5.498425978 podStartE2EDuration="13.287644606s" totalImagesPullingTime="7.789218628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:59.287269527 +0000 UTC m=+12.850732908" watchObservedRunningTime="2026-10-30 19:05:59.287644606 +0000 UTC m=+12.851107984"
Oct 30 19:06:00.039148 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:00.039111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:00.039311 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:00.039252    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:00.241324 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:00.241282    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="7672a8c42cf626344a4deaec11d8becbd028f5a38b5cb9ab65b63bdc33182327" exitCode=0
Oct 30 19:06:00.241489 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:00.241345    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdll" event={"ID":"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb","Type":"ContainerDied","Data":"7672a8c42cf626344a4deaec11d8becbd028f5a38b5cb9ab65b63bdc33182327"}
Oct 30 19:06:01.039223 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:01.039188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:01.039410 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:01.039313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:01.039410 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:01.039362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:01.040025 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:01.039471    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:02.039050 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:02.039018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:02.039241 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:02.039154    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:02.248733 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:02.248699    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="1ac55b136d00118d482a1e118bea33965c64ba627a2d958cba56b644e6ed122d" exitCode=0
Oct 30 19:06:02.249246 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:02.248754    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdll" event={"ID":"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb","Type":"ContainerDied","Data":"1ac55b136d00118d482a1e118bea33965c64ba627a2d958cba56b644e6ed122d"}
Oct 30 19:06:03.040663 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:03.040261    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:03.040663 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:03.040271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:03.040663 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:03.040383    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:03.040663 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:03.040452    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:03.252925 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:03.252894    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" event={"ID":"f31e88aa-14a5-4deb-a503-17b7f52cb4b9","Type":"ContainerStarted","Data":"72c61848a4962a4391c121e3c70b2c4e02c6ea0e4e097b3b1911450119599ff1"}
Oct 30 19:06:03.253378 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:03.252987    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" event={"ID":"f31e88aa-14a5-4deb-a503-17b7f52cb4b9","Type":"ContainerStarted","Data":"3f865d5fc4c767e911cf9377264622fc7a02a9d37eb32d95cbeb6233a09f431c"}
Oct 30 19:06:04.039439 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.039200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:04.039630 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:04.039568    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:04.257869 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.257834    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="cfc6344599fb013241c4c7c55bb8d37bf21f5fc9686277554c4156ebe6a8ba88" exitCode=0
Oct 30 19:06:04.258298 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.257926    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdll" event={"ID":"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb","Type":"ContainerDied","Data":"cfc6344599fb013241c4c7c55bb8d37bf21f5fc9686277554c4156ebe6a8ba88"}
Oct 30 19:06:04.261041 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.260991    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" event={"ID":"f31e88aa-14a5-4deb-a503-17b7f52cb4b9","Type":"ContainerStarted","Data":"7b59ceab515e12f0f2f720aad449a75373b593259aeeba6a095fc4aa78725182"}
Oct 30 19:06:04.261157 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.261094    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" event={"ID":"f31e88aa-14a5-4deb-a503-17b7f52cb4b9","Type":"ContainerStarted","Data":"f523df4f5c60b84bfad01947035d46668f0d69fca96c40cfd6290981807870d9"}
Oct 30 19:06:04.261157 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.261148    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" event={"ID":"f31e88aa-14a5-4deb-a503-17b7f52cb4b9","Type":"ContainerStarted","Data":"3047c1a8882d8cafd193b251f0ab63e6a8c8616678bb577cdf2b9ebf6e4db92f"}
Oct 30 19:06:04.261260 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.261193    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" event={"ID":"f31e88aa-14a5-4deb-a503-17b7f52cb4b9","Type":"ContainerStarted","Data":"f5696c53d3a06da198f522bbd8ddea9ef27c52c49c6346df48bf1d1c5ae09d8a"}
Oct 30 19:06:04.537835 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.537738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:04.537835 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.537809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:04.538084 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:04.537896    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:06:04.538084 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:04.537963    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret podName:baec05dd-6d69-41d9-ad73-4179b17ba8f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:20.537950029 +0000 UTC m=+34.101413390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret") pod "global-pull-secret-syncer-tvwkt" (UID: "baec05dd-6d69-41d9-ad73-4179b17ba8f4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:06:04.538084 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:04.537969    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:06:04.538084 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:04.538017    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs podName:ceeebf2b-9b37-4988-b6c1-b8b41e09449a nodeName:}" failed. No retries permitted until 2026-10-30 19:06:20.538001741 +0000 UTC m=+34.101465103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs") pod "network-metrics-daemon-tbsh8" (UID: "ceeebf2b-9b37-4988-b6c1-b8b41e09449a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:06:04.739462 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:04.739419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:04.739639 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:04.739571    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:06:04.739639 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:04.739591    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:06:04.739639 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:04.739600    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b5px2 for pod openshift-network-diagnostics/network-check-target-k4plb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:06:04.739765 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:04.739648    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2 podName:6e38a783-1270-4703-a97c-cabdbe1c50ce nodeName:}" failed. No retries permitted until 2026-10-30 19:06:20.739636098 +0000 UTC m=+34.303099460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5px2" (UniqueName: "kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2") pod "network-check-target-k4plb" (UID: "6e38a783-1270-4703-a97c-cabdbe1c50ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:06:05.038671 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:05.038641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:05.038671 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:05.038664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:05.038897 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:05.038750    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:05.038897 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:05.038860    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:05.266540 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:05.266500    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="1085d7272173633864c6c822849ecc3645a82ebc6d786f407fda22709e961eec" exitCode=0
Oct 30 19:06:05.266540 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:05.266543    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdll" event={"ID":"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb","Type":"ContainerDied","Data":"1085d7272173633864c6c822849ecc3645a82ebc6d786f407fda22709e961eec"}
Oct 30 19:06:05.275927 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:05.275899    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ljs_1b26e19e-b8e9-496a-bfcb-6cd5add61e6e/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:05.516486 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:05.516452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ljs_1b26e19e-b8e9-496a-bfcb-6cd5add61e6e/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:05.710206 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:05.710167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ljs_1b26e19e-b8e9-496a-bfcb-6cd5add61e6e/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:05.892116 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:05.892013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ljs_1b26e19e-b8e9-496a-bfcb-6cd5add61e6e/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:06.039220 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:06.039183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:06.039391 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:06.039290    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:06.112272 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:06.112233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ljs_1b26e19e-b8e9-496a-bfcb-6cd5add61e6e/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:06.271744 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:06.271670    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-829qt_38ac8869-277b-44e4-a31a-bde13751efd9/dns-node-resolver/0.log"
Oct 30 19:06:06.272136 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:06.271759    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdll" event={"ID":"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb","Type":"ContainerStarted","Data":"a022afc4f206367e3684094352face339e43625e5c39473232ed4fdb55322eb7"}
Oct 30 19:06:06.275011 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:06.274987    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" event={"ID":"f31e88aa-14a5-4deb-a503-17b7f52cb4b9","Type":"ContainerStarted","Data":"5fb6afe02de803712d45473c4d5e202a4bff1a15ac5a7538f6e57985438fa732"}
Oct 30 19:06:06.294823 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:06.294777    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5bdll" podStartSLOduration=8.510457694 podStartE2EDuration="19.294765053s" totalImagesPullingTime="10.784307359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:06.294627153 +0000 UTC m=+19.858090531" watchObservedRunningTime="2026-10-30 19:06:06.294765053 +0000 UTC m=+19.858228430"
Oct 30 19:06:07.039801 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:07.039764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:07.040001 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:07.039921    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:07.040001 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:07.039967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:07.040145 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:07.040094    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:07.470605 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:07.470534    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9b5ds_0b9c7068-7163-4c61-8a7f-5ec789247fa7/node-ca/0.log"
Oct 30 19:06:08.038355 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:08.038158    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:08.038516 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:08.038436    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:08.282209 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:08.282169    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" event={"ID":"f31e88aa-14a5-4deb-a503-17b7f52cb4b9","Type":"ContainerStarted","Data":"d192c42cdab971fb2e953e9cd4ee776d079f93ca0ac0f37479cf2838ebf94cb1"}
Oct 30 19:06:08.282474 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:08.282456    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:08.282533 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:08.282523    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:08.298680 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:08.298612    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:08.313506 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:08.313459    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8" podStartSLOduration=7.4502210269999996 podStartE2EDuration="21.313446745s" totalImagesPullingTime="13.863225718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:08.313106013 +0000 UTC m=+21.876569390" watchObservedRunningTime="2026-10-30 19:06:08.313446745 +0000 UTC m=+21.876910119"
Oct 30 19:06:09.039207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:09.039165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:09.039207 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:09.039203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:09.039849 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:09.039281    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:09.039849 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:09.039391    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:09.285002 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:09.284971    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:09.300175 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:09.300091    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:10.038391 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:10.038352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:10.038643 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:10.038461    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:11.038771 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:11.038736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:11.038771 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:11.038762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:11.039240 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:11.038850    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:11.039240 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:11.038959    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:12.038575 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:12.038537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:12.038774 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:12.038649    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:13.038477 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:13.038436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:13.038940 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:13.038550    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:13.038940 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:13.038600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:13.038940 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:13.038686    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:14.038481 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:14.038446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:14.038886 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:14.038555    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:15.038415 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:15.038381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:15.038579 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:15.038478    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:15.038579 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:15.038552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:15.038869 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:15.038635    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:16.038356 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:16.038319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:16.038530 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:16.038427    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:17.039336 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:17.039296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:17.039800 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:17.039434    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:17.039800 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:17.039474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:17.039800 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:17.039531    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:18.038369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:18.038331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:18.038546 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:18.038447    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:19.040919 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:19.040893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:19.041366 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:19.040869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:19.041366 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:19.041016    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:19.041494 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:19.041371    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:20.038285 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:20.038254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:20.038462 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:20.038371    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:20.546802 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:20.546776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:20.547263 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:20.546823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:20.547263 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:20.546915    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:06:20.547263 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:20.546934    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:06:20.547263 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:20.546985    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs podName:ceeebf2b-9b37-4988-b6c1-b8b41e09449a nodeName:}" failed. No retries permitted until 2026-10-30 19:06:52.546954758 +0000 UTC m=+66.110418114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs") pod "network-metrics-daemon-tbsh8" (UID: "ceeebf2b-9b37-4988-b6c1-b8b41e09449a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:06:20.547263 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:20.547014    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret podName:baec05dd-6d69-41d9-ad73-4179b17ba8f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:52.546997703 +0000 UTC m=+66.110461065 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret") pod "global-pull-secret-syncer-tvwkt" (UID: "baec05dd-6d69-41d9-ad73-4179b17ba8f4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:06:20.747788 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:20.747753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:20.747974 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:20.747934    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:06:20.747974 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:20.747956    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:06:20.747974 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:20.747967    2578 projected.go:196] Error preparing data for projected volume kube-api-access-b5px2 for pod openshift-network-diagnostics/network-check-target-k4plb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:06:20.748177 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:20.748032    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2 podName:6e38a783-1270-4703-a97c-cabdbe1c50ce nodeName:}" failed. No retries permitted until 2026-10-30 19:06:52.748011874 +0000 UTC m=+66.311475248 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5px2" (UniqueName: "kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2") pod "network-check-target-k4plb" (UID: "6e38a783-1270-4703-a97c-cabdbe1c50ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:06:21.038796 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:21.038766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:21.038983 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:21.038828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:21.038983 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:21.038945    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:21.039111 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:21.039037    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:22.038302 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:22.038263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:22.038688 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:22.038379    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:23.038515 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:23.038479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:23.038958 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:23.038483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:23.038958 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:23.038621    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:23.038958 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:23.038689    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:23.457786 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:23.457707    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k4plb"]
Oct 30 19:06:23.457928 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:23.457883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:23.458012 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:23.457987    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:23.458940 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:23.458853    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tvwkt"]
Oct 30 19:06:23.459142 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:23.459000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:23.459142 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:23.459106    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:23.460199 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:23.460163    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tbsh8"]
Oct 30 19:06:23.460292 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:23.460285    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:23.460518 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:23.460383    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:24.232969 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:24.232895    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-829qt_38ac8869-277b-44e4-a31a-bde13751efd9/dns-node-resolver/0.log"
Oct 30 19:06:25.039215 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:25.039188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:25.039215 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:25.039203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:25.039437 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:25.039306    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:25.039437 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:25.039349    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:25.039437 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:25.039357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:25.039530 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:25.039438    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:25.632693 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:25.632663    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9b5ds_0b9c7068-7163-4c61-8a7f-5ec789247fa7/node-ca/0.log"
Oct 30 19:06:27.039426 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:27.039236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:27.039880 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:27.039314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:27.039880 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:27.039509    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k4plb" podUID="6e38a783-1270-4703-a97c-cabdbe1c50ce"
Oct 30 19:06:27.039880 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:27.039338    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:27.039880 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:27.039620    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tbsh8" podUID="ceeebf2b-9b37-4988-b6c1-b8b41e09449a"
Oct 30 19:06:27.039880 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:27.039715    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvwkt" podUID="baec05dd-6d69-41d9-ad73-4179b17ba8f4"
Oct 30 19:06:28.235911 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.235841    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeReady"
Oct 30 19:06:28.236317 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.235967    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:06:28.287547 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.287521    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vpj4h"]
Oct 30 19:06:28.320157 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.320129    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m89p4"]
Oct 30 19:06:28.320332 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.320252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.329223 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.328742    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6h85f\""
Oct 30 19:06:28.329223 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.328817    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:06:28.329359 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.329342    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:06:28.334108 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.334088    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m89p4"]
Oct 30 19:06:28.334218 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.334124    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpj4h"]
Oct 30 19:06:28.334218 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.334159    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vtpvd"]
Oct 30 19:06:28.334321 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.334276    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:28.345962 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.345945    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:06:28.346130 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.346117    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:06:28.347370 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.347353    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vtpvd"]
Oct 30 19:06:28.347462 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.347399    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f645b69cc-d5dm6"]
Oct 30 19:06:28.347526 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.347513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.361523 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.361500    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f645b69cc-d5dm6"]
Oct 30 19:06:28.361523 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.361519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.429368 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.429335    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:06:28.429587 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.429568    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:06:28.429688 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.429670    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 19:06:28.429732 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.429671    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 19:06:28.429770 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.429755    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:06:28.429813 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.429790    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xxr6f\""
Oct 30 19:06:28.430122 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.430097    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:06:28.430234 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.430098    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:06:28.430234 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.430141    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:06:28.430234 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.430117    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 19:06:28.430234 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.430162    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fd7q8\""
Oct 30 19:06:28.430234 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.430119    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 19:06:28.430234 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.430105    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 19:06:28.430234 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.430207    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fsc2n\""
Oct 30 19:06:28.430234 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.430212    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:06:28.506663 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.506586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60953151-cd1e-4572-a9f1-a618fee9ef83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.506663 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.506627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2nnm\" (UniqueName: \"kubernetes.io/projected/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-kube-api-access-b2nnm\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:28.506663 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.506643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrmg\" (UniqueName: \"kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.506895 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.506728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.506895 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.506855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-cert\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:28.506895 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.506886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.507039 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.506911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.507039 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.506953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a979a593-d7fc-4659-a932-ca61a3801f27-config-volume\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.507039 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.506998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a979a593-d7fc-4659-a932-ca61a3801f27-metrics-tls\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.507039 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.507015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60953151-cd1e-4572-a9f1-a618fee9ef83-crio-socket\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.507210 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.507104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.507210 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.507129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.507210 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.507205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrl6\" (UniqueName: \"kubernetes.io/projected/a979a593-d7fc-4659-a932-ca61a3801f27-kube-api-access-frrl6\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.507299 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.507229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.507299 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.507264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvjr\" (UniqueName: \"kubernetes.io/projected/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-api-access-qhvjr\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.507299 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.507294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.507406 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.507312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60953151-cd1e-4572-a9f1-a618fee9ef83-data-volume\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.507406 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.507346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a979a593-d7fc-4659-a932-ca61a3801f27-tmp-dir\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.608126 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-cert\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:28.608210 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.608210 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.608210 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a979a593-d7fc-4659-a932-ca61a3801f27-config-volume\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.608210 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a979a593-d7fc-4659-a932-ca61a3801f27-metrics-tls\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.608210 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60953151-cd1e-4572-a9f1-a618fee9ef83-crio-socket\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrl6\" (UniqueName: \"kubernetes.io/projected/a979a593-d7fc-4659-a932-ca61a3801f27-kube-api-access-frrl6\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvjr\" (UniqueName: \"kubernetes.io/projected/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-api-access-qhvjr\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60953151-cd1e-4572-a9f1-a618fee9ef83-data-volume\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a979a593-d7fc-4659-a932-ca61a3801f27-tmp-dir\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60953151-cd1e-4572-a9f1-a618fee9ef83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2nnm\" (UniqueName: \"kubernetes.io/projected/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-kube-api-access-b2nnm\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608474    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60953151-cd1e-4572-a9f1-a618fee9ef83-crio-socket\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrmg\" (UniqueName: \"kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.608561 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.608545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.609160 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.609091    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a979a593-d7fc-4659-a932-ca61a3801f27-config-volume\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.609300 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.609279    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.609464 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.609440    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.609603 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.609584    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.609730 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.609591    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60953151-cd1e-4572-a9f1-a618fee9ef83-data-volume\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.609827 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.609805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a979a593-d7fc-4659-a932-ca61a3801f27-tmp-dir\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.609905 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.609619    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.609977 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.609956    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.612752 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.612717    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a979a593-d7fc-4659-a932-ca61a3801f27-metrics-tls\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.612752 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.612729    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.612752 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.612719    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60953151-cd1e-4572-a9f1-a618fee9ef83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.612950 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.612719    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-cert\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:28.612950 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.612778    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.619807 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.619784    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvjr\" (UniqueName: \"kubernetes.io/projected/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-api-access-qhvjr\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:28.619925 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.619854    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrl6\" (UniqueName: \"kubernetes.io/projected/a979a593-d7fc-4659-a932-ca61a3801f27-kube-api-access-frrl6\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.620636 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.620616    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2nnm\" (UniqueName: \"kubernetes.io/projected/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-kube-api-access-b2nnm\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:28.620929 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.620909    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrmg\" (UniqueName: \"kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.661849 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.661808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:28.690297 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.690268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:28.704438 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.704409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:28.732220 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:28.732191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:29.001392 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.001359    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m89p4"]
Oct 30 19:06:29.008821 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.008108    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpj4h"]
Oct 30 19:06:29.012205 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:29.012167    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda979a593_d7fc_4659_a932_ca61a3801f27.slice/crio-207fd225e3156d80126edea5098e3657deb390fe9a23f6cbdd7322fd82d844d3 WatchSource:0}: Error finding container 207fd225e3156d80126edea5098e3657deb390fe9a23f6cbdd7322fd82d844d3: Status 404 returned error can't find the container with id 207fd225e3156d80126edea5098e3657deb390fe9a23f6cbdd7322fd82d844d3
Oct 30 19:06:29.022593 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.022572    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f645b69cc-d5dm6"]
Oct 30 19:06:29.024355 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:29.024318    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ebd094_601c_462b_a42c_89e9d6a27c2a.slice/crio-93ee2a35bd62881dac05f034f08a6d150907923921c956046f8846c0803ae693 WatchSource:0}: Error finding container 93ee2a35bd62881dac05f034f08a6d150907923921c956046f8846c0803ae693: Status 404 returned error can't find the container with id 93ee2a35bd62881dac05f034f08a6d150907923921c956046f8846c0803ae693
Oct 30 19:06:29.038978 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.038926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:29.039159 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.038991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:29.039159 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.039007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:29.043357 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.043153    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:06:29.043488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.043161    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:06:29.043488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.043161    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:06:29.043488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.043212    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-89jvk\""
Oct 30 19:06:29.043488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.043245    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tc7v2\""
Oct 30 19:06:29.043488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.043260    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fljvz\""
Oct 30 19:06:29.043488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.043299    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:06:29.044457 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.044437    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vtpvd"]
Oct 30 19:06:29.047634 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:29.047606    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60953151_cd1e_4572_a9f1_a618fee9ef83.slice/crio-157faf72a653c5be0d1f54bc2bc0d127aef12da8638b660a434e0ae5072b2b4b WatchSource:0}: Error finding container 157faf72a653c5be0d1f54bc2bc0d127aef12da8638b660a434e0ae5072b2b4b: Status 404 returned error can't find the container with id 157faf72a653c5be0d1f54bc2bc0d127aef12da8638b660a434e0ae5072b2b4b
Oct 30 19:06:29.329275 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.329238    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vtpvd" event={"ID":"60953151-cd1e-4572-a9f1-a618fee9ef83","Type":"ContainerStarted","Data":"175648511edcd9352842bdf7621c91645134dc445017c5a4a447780ea724c4e9"}
Oct 30 19:06:29.330104 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.329304    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vtpvd" event={"ID":"60953151-cd1e-4572-a9f1-a618fee9ef83","Type":"ContainerStarted","Data":"157faf72a653c5be0d1f54bc2bc0d127aef12da8638b660a434e0ae5072b2b4b"}
Oct 30 19:06:29.330425 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.330403    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpj4h" event={"ID":"a979a593-d7fc-4659-a932-ca61a3801f27","Type":"ContainerStarted","Data":"207fd225e3156d80126edea5098e3657deb390fe9a23f6cbdd7322fd82d844d3"}
Oct 30 19:06:29.331461 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.331435    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m89p4" event={"ID":"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3","Type":"ContainerStarted","Data":"d761acb566f985e1207ae0b8637b8299da6f7f44b89cc59394a762c905c3c399"}
Oct 30 19:06:29.332546 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:29.332527    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f645b69cc-d5dm6" event={"ID":"f2ebd094-601c-462b-a42c-89e9d6a27c2a","Type":"ContainerStarted","Data":"93ee2a35bd62881dac05f034f08a6d150907923921c956046f8846c0803ae693"}
Oct 30 19:06:31.346655 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:31.346569    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vtpvd" event={"ID":"60953151-cd1e-4572-a9f1-a618fee9ef83","Type":"ContainerStarted","Data":"540395ec04241eedecb5dbfc3e71b9510f30e99a3fcd2bb4c222f6b4b1bc3a03"}
Oct 30 19:06:32.088203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.088126    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"]
Oct 30 19:06:32.096151 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.095766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.113103 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.112950    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 19:06:32.114445 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.113857    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"]
Oct 30 19:06:32.114445 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.114085    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:06:32.114445 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.114170    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:06:32.114445 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.114244    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:06:32.114445 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.114299    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:06:32.114445 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.114365    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mzlzt\""
Oct 30 19:06:32.124965 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.124940    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"]
Oct 30 19:06:32.125107 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.124983    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tsr7c"]
Oct 30 19:06:32.125702 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.125204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.143137 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143041    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"]
Oct 30 19:06:32.143256 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.143321 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdgm\" (UniqueName: \"kubernetes.io/projected/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-api-access-5wdgm\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.143321 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61317c8f-5a2e-4268-b085-df795ec4baa6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.143321 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8km5\" (UniqueName: \"kubernetes.io/projected/61317c8f-5a2e-4268-b085-df795ec4baa6-kube-api-access-l8km5\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.143583 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/558fbbeb-c828-4767-aecc-8c319ab91b35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.143583 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.143583 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.143583 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.143583 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.143583 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.143891 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.143592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.165049 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.165014    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:06:32.165283 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.165258    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 19:06:32.165412 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.165309    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:06:32.165412 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.165350    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n985b\""
Oct 30 19:06:32.165412 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.165401    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wdms4\""
Oct 30 19:06:32.165569 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.165430    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:06:32.204889 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.204856    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:06:32.205111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.204934    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 19:06:32.245159 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-tls\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.245308 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.245354 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/558fbbeb-c828-4767-aecc-8c319ab91b35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.245418 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-root\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.245418 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.245488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.245531 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.245585 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-wtmp\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.245710 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.245765 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.245765 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-metrics-client-ca\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.245978 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.245961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdgm\" (UniqueName: \"kubernetes.io/projected/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-api-access-5wdgm\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.246045 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.246002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61317c8f-5a2e-4268-b085-df795ec4baa6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.246045 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.246028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8km5\" (UniqueName: \"kubernetes.io/projected/61317c8f-5a2e-4268-b085-df795ec4baa6-kube-api-access-l8km5\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.246247 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.246231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9p5\" (UniqueName: \"kubernetes.io/projected/91513898-3858-4991-afad-cb5999ab98d0-kube-api-access-hg9p5\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.246312 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.246265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.256082 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.246305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.256082 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.246470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-textfile\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.256082 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.246518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-sys\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.256082 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.255281    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/558fbbeb-c828-4767-aecc-8c319ab91b35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.256082 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.255576    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.256454 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.255973    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61317c8f-5a2e-4268-b085-df795ec4baa6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.257094 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.256757    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.262544 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.259630    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.262544 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.260209    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.262544 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.262502    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.268567 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.268500    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.274369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.274156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdgm\" (UniqueName: \"kubernetes.io/projected/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-api-access-5wdgm\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.274538 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.274518    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8km5\" (UniqueName: \"kubernetes.io/projected/61317c8f-5a2e-4268-b085-df795ec4baa6-kube-api-access-l8km5\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.347254 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-metrics-client-ca\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.347716 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9p5\" (UniqueName: \"kubernetes.io/projected/91513898-3858-4991-afad-cb5999ab98d0-kube-api-access-hg9p5\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.347716 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-textfile\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.347716 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-sys\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.347716 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-tls\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.347716 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.347716 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-root\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.347716 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.347716 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-wtmp\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.347716 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.347687    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-wtmp\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.348318 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.348290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-metrics-client-ca\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.348874 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.348853    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-textfile\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.348973 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.348924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-sys\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.349792 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.349156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-root\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.349792 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.349745    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.351574 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.351552    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-tls\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.351802 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.351775    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.354834 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.354808    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpj4h" event={"ID":"a979a593-d7fc-4659-a932-ca61a3801f27","Type":"ContainerStarted","Data":"6e4cab86a6b83b9adab76834b2e48adf07f5f3cf2c1b5342afac727ec0b2bfa8"}
Oct 30 19:06:32.354942 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.354890    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpj4h" event={"ID":"a979a593-d7fc-4659-a932-ca61a3801f27","Type":"ContainerStarted","Data":"b3a27f290b6ae360b8aea80fcc5ec930cb0b4c964534459e992535f6e53e870a"}
Oct 30 19:06:32.355085 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.355011    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:32.362785 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.362756    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9p5\" (UniqueName: \"kubernetes.io/projected/91513898-3858-4991-afad-cb5999ab98d0-kube-api-access-hg9p5\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.374898 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.374790    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vpj4h" podStartSLOduration=1.5200402290000001 podStartE2EDuration="4.374774017s" totalImagesPullingTime="2.854733788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:32.374036607 +0000 UTC m=+45.937499986" watchObservedRunningTime="2026-10-30 19:06:32.374774017 +0000 UTC m=+45.938237396"
Oct 30 19:06:32.466556 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.466522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:32.511720 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.511688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:32.527687 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.527502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:32.534140 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:32.534114    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91513898_3858_4991_afad_cb5999ab98d0.slice/crio-59f5ff61650f5215af4d4b0ae16aa6ddea87d2faabadaa6b06a368007b356490 WatchSource:0}: Error finding container 59f5ff61650f5215af4d4b0ae16aa6ddea87d2faabadaa6b06a368007b356490: Status 404 returned error can't find the container with id 59f5ff61650f5215af4d4b0ae16aa6ddea87d2faabadaa6b06a368007b356490
Oct 30 19:06:32.769798 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.769765    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"]
Oct 30 19:06:32.842326 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:32.842286    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61317c8f_5a2e_4268_b085_df795ec4baa6.slice/crio-ea79321c3382399cb3ef51cb5674f9dd0b1173c18bd1ee014e70008fdec4f4ea WatchSource:0}: Error finding container ea79321c3382399cb3ef51cb5674f9dd0b1173c18bd1ee014e70008fdec4f4ea: Status 404 returned error can't find the container with id ea79321c3382399cb3ef51cb5674f9dd0b1173c18bd1ee014e70008fdec4f4ea
Oct 30 19:06:32.913466 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:32.913432    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"]
Oct 30 19:06:33.054938 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.054902    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:06:33.074675 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.074652    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.078741 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.078699    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:06:33.079283 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079259    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:06:33.079425 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079299    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:06:33.079425 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079350    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:06:33.079425 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079379    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:06:33.079425 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079412    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:06:33.079425 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079299    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:06:33.079644 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079439    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:06:33.079644 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079486    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sg4ml\""
Oct 30 19:06:33.079644 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079523    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:06:33.079644 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.079534    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:06:33.154443 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154443 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154664 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154664 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154664 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgsn\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154664 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154664 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154664 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154912 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154912 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154912 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154912 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.154912 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.154896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.255325 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.255297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.255479 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.255329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.255479 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.255371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.255479 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.255473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.255634 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.255495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.255634 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.255515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.255634 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.255537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.255634 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.255563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.255634 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.255607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.256263 ip-10-0-132-19 kubenswrapper[2578]: E1030 19:06:33.255992    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle podName:09ea8db5-8a84-4f64-a520-8089c8ee8231 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:33.755955753 +0000 UTC m=+47.319419130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231") : configmap references non-existent config key: ca-bundle.crt
Oct 30 19:06:33.256263 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.256125    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.256522 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.256446    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.256908 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.256871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgsn\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.257011 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.256926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.257011 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.256992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.257122 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.257019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.259488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.259016    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.259488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.259224    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.259488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.259445    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.259687 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.259532    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.259687 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.259621    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.260475 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.260450    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.261676 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.261649    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.261828 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.261813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.262254 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.262236    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.269549 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.269531    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgsn\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.360807 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.360762    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m89p4" event={"ID":"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3","Type":"ContainerStarted","Data":"22e66acb0f18cb3e3340859111b515cc3aa7a5aab467ec167c251dc92944e9a1"}
Oct 30 19:06:33.362988 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.362936    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn" event={"ID":"61317c8f-5a2e-4268-b085-df795ec4baa6","Type":"ContainerStarted","Data":"858e0ab09518a49ab2d487fadcb35d4298744de8cef821324fb4bd2e62acae42"}
Oct 30 19:06:33.363113 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.363009    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn" event={"ID":"61317c8f-5a2e-4268-b085-df795ec4baa6","Type":"ContainerStarted","Data":"38082badb5ab262275600c87ff4ddda0e553f353ca09be1f629758d80a367dba"}
Oct 30 19:06:33.363113 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.363036    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn" event={"ID":"61317c8f-5a2e-4268-b085-df795ec4baa6","Type":"ContainerStarted","Data":"ea79321c3382399cb3ef51cb5674f9dd0b1173c18bd1ee014e70008fdec4f4ea"}
Oct 30 19:06:33.364766 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.364741    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f645b69cc-d5dm6" event={"ID":"f2ebd094-601c-462b-a42c-89e9d6a27c2a","Type":"ContainerStarted","Data":"9275219a3681eaffee601a87bf8132bab83bae839d06f67733a8cdadc14ec4db"}
Oct 30 19:06:33.367329 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.367302    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vtpvd" event={"ID":"60953151-cd1e-4572-a9f1-a618fee9ef83","Type":"ContainerStarted","Data":"679855db77dc9d98a8a2195fdd30c343209ce855d5cf9a6630b1a54dd83b2309"}
Oct 30 19:06:33.368685 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.368662    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs" event={"ID":"558fbbeb-c828-4767-aecc-8c319ab91b35","Type":"ContainerStarted","Data":"1190742f738325737b9af75fe35985b2684552445fbda1a511d12a0882cb3f15"}
Oct 30 19:06:33.369899 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.369875    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsr7c" event={"ID":"91513898-3858-4991-afad-cb5999ab98d0","Type":"ContainerStarted","Data":"59f5ff61650f5215af4d4b0ae16aa6ddea87d2faabadaa6b06a368007b356490"}
Oct 30 19:06:33.393980 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.393923    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m89p4" podStartSLOduration=1.7559763259999999 podStartE2EDuration="5.393910999s" totalImagesPullingTime="3.637934673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:33.393587864 +0000 UTC m=+46.957051242" watchObservedRunningTime="2026-10-30 19:06:33.393910999 +0000 UTC m=+46.957374391"
Oct 30 19:06:33.420341 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.420242    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vtpvd" podStartSLOduration=1.802031361 podStartE2EDuration="5.420228795s" totalImagesPullingTime="3.618197434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:06:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:33.419179679 +0000 UTC m=+46.982643058" watchObservedRunningTime="2026-10-30 19:06:33.420228795 +0000 UTC m=+46.983692194"
Oct 30 19:06:33.762271 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.762091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.763083 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.763032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:33.798570 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:33.798534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:34.080265 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.080199    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f645b69cc-d5dm6" podStartSLOduration=1.890880702 podStartE2EDuration="6.080178373s" totalImagesPullingTime="4.189297671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:33.440487353 +0000 UTC m=+47.003950735" watchObservedRunningTime="2026-10-30 19:06:34.080178373 +0000 UTC m=+47.643641752"
Oct 30 19:06:34.081286 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.081265    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77856dc4d7-j9gps"]
Oct 30 19:06:34.084101 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.084080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.091252 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.091201    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f6l0lstb29pus\""
Oct 30 19:06:34.091252 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.091215    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 19:06:34.091252 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.091211    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 19:06:34.091252 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.091211    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 19:06:34.091252 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.091248    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 19:06:34.091516 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.091295    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9ctdn\""
Oct 30 19:06:34.091516 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.091297    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 19:06:34.098305 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.098283    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77856dc4d7-j9gps"]
Oct 30 19:06:34.107149 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.107124    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:06:34.108654 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:34.108599    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ea8db5_8a84_4f64_a520_8089c8ee8231.slice/crio-91ec2d72013cd4defd209ccca364d3c6550f52a3e42234c7a8772b296a560ee0 WatchSource:0}: Error finding container 91ec2d72013cd4defd209ccca364d3c6550f52a3e42234c7a8772b296a560ee0: Status 404 returned error can't find the container with id 91ec2d72013cd4defd209ccca364d3c6550f52a3e42234c7a8772b296a560ee0
Oct 30 19:06:34.164493 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.164463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.164594 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.164542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswgb\" (UniqueName: \"kubernetes.io/projected/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-kube-api-access-fswgb\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.164594 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.164573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.164674 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.164603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-grpc-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.164736 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.164721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-metrics-client-ca\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.164784 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.164745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.164784 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.164765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-tmp\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.164871 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.164857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.164920 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.164884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.266926 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.266873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.267150 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.266991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswgb\" (UniqueName: \"kubernetes.io/projected/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-kube-api-access-fswgb\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.267150 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.267026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.267308 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.267284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-grpc-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.267369 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.267354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-metrics-client-ca\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.267415 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.267386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.267466 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.267421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-tmp\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.267519 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.267489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.267567 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.267524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.268089 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.268042    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-metrics-client-ca\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.268206 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.268107    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-tmp\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.269724 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.269696    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.270443 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.270389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.270601 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.270562    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.270701 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.270685    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-grpc-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.271107 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.271086    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.271471 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.271452    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.282358 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.282339    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswgb\" (UniqueName: \"kubernetes.io/projected/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-kube-api-access-fswgb\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.378049 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.378016    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn" event={"ID":"61317c8f-5a2e-4268-b085-df795ec4baa6","Type":"ContainerStarted","Data":"0bcb58444b40a5cf7bf930114f5ab0c72fe2e677c73f54cfd0b8811f1c0d3634"}
Oct 30 19:06:34.379309 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.379284    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerStarted","Data":"91ec2d72013cd4defd209ccca364d3c6550f52a3e42234c7a8772b296a560ee0"}
Oct 30 19:06:34.380963 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.380931    2578 generic.go:356] "Generic (PLEG): container finished" podID="91513898-3858-4991-afad-cb5999ab98d0" containerID="960242d5933e70aa8b921be6625d707285a1beebe2c3b36cf99c1aee2b7a5a9a" exitCode=0
Oct 30 19:06:34.381088 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.381013    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsr7c" event={"ID":"91513898-3858-4991-afad-cb5999ab98d0","Type":"ContainerDied","Data":"960242d5933e70aa8b921be6625d707285a1beebe2c3b36cf99c1aee2b7a5a9a"}
Oct 30 19:06:34.396749 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.396705    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn" podStartSLOduration=1.3270496920000001 podStartE2EDuration="2.396692988s" totalImagesPullingTime="1.069643296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:34.396265654 +0000 UTC m=+47.959729033" watchObservedRunningTime="2026-10-30 19:06:34.396692988 +0000 UTC m=+47.960156366"
Oct 30 19:06:34.410685 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.410666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:34.699860 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:34.699778    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77856dc4d7-j9gps"]
Oct 30 19:06:34.702529 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:34.702490    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e0ef42_c5a6_43d5_8eb7_92f65fa2d634.slice/crio-c8a85eb6caf3a654f8e227d9eb18c7c2e526ad1319967a86fd8aa37742188649 WatchSource:0}: Error finding container c8a85eb6caf3a654f8e227d9eb18c7c2e526ad1319967a86fd8aa37742188649: Status 404 returned error can't find the container with id c8a85eb6caf3a654f8e227d9eb18c7c2e526ad1319967a86fd8aa37742188649
Oct 30 19:06:35.387373 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.387333    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs" event={"ID":"558fbbeb-c828-4767-aecc-8c319ab91b35","Type":"ContainerStarted","Data":"e69afcff1a1062c6d39c679d8c587445124a8b864756d150f302e30b0348e194"}
Oct 30 19:06:35.387980 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.387406    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs" event={"ID":"558fbbeb-c828-4767-aecc-8c319ab91b35","Type":"ContainerStarted","Data":"b37a862fd0b9e285f3d5e7a2b8a85ff8965fd2b7986264672c42cff1e848a549"}
Oct 30 19:06:35.387980 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.387430    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs" event={"ID":"558fbbeb-c828-4767-aecc-8c319ab91b35","Type":"ContainerStarted","Data":"91cf4abb4e11f706f0d38bfaa198893140aca4322e13b4f20941d01545b0536b"}
Oct 30 19:06:35.390260 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.390231    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsr7c" event={"ID":"91513898-3858-4991-afad-cb5999ab98d0","Type":"ContainerStarted","Data":"a1a2b406dbb8db75166d9224a4113b4c9ea872d3b9895c9e19bd3d5df532f303"}
Oct 30 19:06:35.390423 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.390294    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsr7c" event={"ID":"91513898-3858-4991-afad-cb5999ab98d0","Type":"ContainerStarted","Data":"869a64fa2a9324b807e17afa5fda03d8a00ada13f379d5aec88d7ff08f2ba02c"}
Oct 30 19:06:35.391770 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.391744    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps" event={"ID":"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634","Type":"ContainerStarted","Data":"c8a85eb6caf3a654f8e227d9eb18c7c2e526ad1319967a86fd8aa37742188649"}
Oct 30 19:06:35.393521 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.393499    2578 generic.go:356] "Generic (PLEG): container finished" podID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerID="477c9e680d4f90899fc2330471567ed48624dd5e6a60b5216daab4e08a9a50f4" exitCode=0
Oct 30 19:06:35.393682 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.393592    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerDied","Data":"477c9e680d4f90899fc2330471567ed48624dd5e6a60b5216daab4e08a9a50f4"}
Oct 30 19:06:35.412465 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.412415    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs" podStartSLOduration=1.989780503 podStartE2EDuration="3.412401908s" totalImagesPullingTime="1.422621405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:35.411544294 +0000 UTC m=+48.975007674" watchObservedRunningTime="2026-10-30 19:06:35.412401908 +0000 UTC m=+48.975865284"
Oct 30 19:06:35.438894 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:35.438838    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tsr7c" podStartSLOduration=2.197475656 podStartE2EDuration="3.438819094s" totalImagesPullingTime="1.165439499s" totalInitContainerRuntime="75.903939ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:35.436909716 +0000 UTC m=+49.000373094" watchObservedRunningTime="2026-10-30 19:06:35.438819094 +0000 UTC m=+49.002282474"
Oct 30 19:06:36.500034 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.499999    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-665cc87487-rzd5l"]
Oct 30 19:06:36.503403 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.502587    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.507118 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.507035    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 19:06:36.507118 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.507081    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 19:06:36.518590 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.518523    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-665cc87487-rzd5l"]
Oct 30 19:06:36.531550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.531040    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 19:06:36.531550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.531205    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ams4o7htr81ts\""
Oct 30 19:06:36.531550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.531342    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bbsbt\""
Oct 30 19:06:36.531890 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.531849    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 19:06:36.591145 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.591108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-metrics-server-audit-profiles\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.591315 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.591233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.591315 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.591274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7vwl\" (UniqueName: \"kubernetes.io/projected/f3925cae-4e22-41e0-b510-7c5cec2bfd17-kube-api-access-l7vwl\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.591428 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.591332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-tls\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.591428 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.591379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-client-certs\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.591428 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.591396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f3925cae-4e22-41e0-b510-7c5cec2bfd17-audit-log\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.591428 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.591414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-client-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.692427 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.692395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.692541 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.692446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7vwl\" (UniqueName: \"kubernetes.io/projected/f3925cae-4e22-41e0-b510-7c5cec2bfd17-kube-api-access-l7vwl\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.692541 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.692484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-tls\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.692652 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.692540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-client-certs\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.692652 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.692563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f3925cae-4e22-41e0-b510-7c5cec2bfd17-audit-log\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.694677 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.693351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-client-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.694677 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.693450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-metrics-server-audit-profiles\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.694677 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.694213    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.694677 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.694520    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f3925cae-4e22-41e0-b510-7c5cec2bfd17-audit-log\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.694677 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.694634    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-metrics-server-audit-profiles\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.695869 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.695845    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-tls\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.698040 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.697997    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-client-certs\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.698154 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.698079    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-client-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.705973 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.705942    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7vwl\" (UniqueName: \"kubernetes.io/projected/f3925cae-4e22-41e0-b510-7c5cec2bfd17-kube-api-access-l7vwl\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.780855 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.780828    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"]
Oct 30 19:06:36.784096 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.783861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:36.792149 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.792124    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 19:06:36.792267 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.792145    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7bpwj\""
Oct 30 19:06:36.794619 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.794600    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"]
Oct 30 19:06:36.816344 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.816319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:36.895117 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.895089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r27l\" (UniqueName: \"kubernetes.io/projected/77ff1252-ac70-4fad-a824-99f3c1eb860e-kube-api-access-6r27l\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:36.895195 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.895182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77ff1252-ac70-4fad-a824-99f3c1eb860e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:36.996210 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.996181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r27l\" (UniqueName: \"kubernetes.io/projected/77ff1252-ac70-4fad-a824-99f3c1eb860e-kube-api-access-6r27l\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:36.996305 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.996263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77ff1252-ac70-4fad-a824-99f3c1eb860e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:36.998809 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:36.998791    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77ff1252-ac70-4fad-a824-99f3c1eb860e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:37.008365 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.008348    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r27l\" (UniqueName: \"kubernetes.io/projected/77ff1252-ac70-4fad-a824-99f3c1eb860e-kube-api-access-6r27l\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:37.113698 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.113669    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-665cc87487-rzd5l"]
Oct 30 19:06:37.117859 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:37.117836    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3925cae_4e22_41e0_b510_7c5cec2bfd17.slice/crio-6136986741e80d17c0f61247a519e3761e3afb88427f9b341bf05ebef856b633 WatchSource:0}: Error finding container 6136986741e80d17c0f61247a519e3761e3afb88427f9b341bf05ebef856b633: Status 404 returned error can't find the container with id 6136986741e80d17c0f61247a519e3761e3afb88427f9b341bf05ebef856b633
Oct 30 19:06:37.126858 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.126759    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:37.396810 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.396788    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"]
Oct 30 19:06:37.400007 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:37.399969    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ff1252_ac70_4fad_a824_99f3c1eb860e.slice/crio-cf809d055ebd465e6ddc832e307eac06e6d7a25653f299b341f3b92b3e29cf9c WatchSource:0}: Error finding container cf809d055ebd465e6ddc832e307eac06e6d7a25653f299b341f3b92b3e29cf9c: Status 404 returned error can't find the container with id cf809d055ebd465e6ddc832e307eac06e6d7a25653f299b341f3b92b3e29cf9c
Oct 30 19:06:37.405450 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.405424    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps" event={"ID":"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634","Type":"ContainerStarted","Data":"c443b2c3c0ffe717433e30e2aa0e391938a8c0e27b67cb99f62f35c8b8763ce9"}
Oct 30 19:06:37.405537 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.405504    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps" event={"ID":"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634","Type":"ContainerStarted","Data":"0410375689d6f0268c8ca7df1068331a0dce758cea1d77ec1ba6aa11181dcf06"}
Oct 30 19:06:37.405603 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.405545    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps" event={"ID":"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634","Type":"ContainerStarted","Data":"75d242d9050269a39366f999634725875c12cbf68f583417eb54ae6425d77cce"}
Oct 30 19:06:37.406940 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.406912    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-665cc87487-rzd5l" event={"ID":"f3925cae-4e22-41e0-b510-7c5cec2bfd17","Type":"ContainerStarted","Data":"6136986741e80d17c0f61247a519e3761e3afb88427f9b341bf05ebef856b633"}
Oct 30 19:06:37.410460 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.410430    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerStarted","Data":"91696330db95516eba6dc9dd915b57b425c7232ed18bf3910a9597d4e7f42770"}
Oct 30 19:06:37.410550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.410504    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerStarted","Data":"d3495f64628493e7d3741f8321cca49762e390ab26564d8df44c3d089a349490"}
Oct 30 19:06:37.410550 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.410533    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerStarted","Data":"63257b9f7a0346d3bf02b9d3cf07e83134f06fb36f0a62233ca557017218890d"}
Oct 30 19:06:37.410626 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.410562    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerStarted","Data":"818d8e2701017f58ea2fd0a9cf8b328205d5db465ccd3af91d5d8fc0afc7f017"}
Oct 30 19:06:37.410626 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:37.410584    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerStarted","Data":"30cff2d19c3b93498e05f0af04a666158378dc198da2853aaae5b934feabb715"}
Oct 30 19:06:38.351885 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.351851    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6474bf5b46-vc9zh"]
Oct 30 19:06:38.354400 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.354376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.367833 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.367715    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 19:06:38.371020 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.370661    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:06:38.373900 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.373873    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6474bf5b46-vc9zh"]
Oct 30 19:06:38.374017 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.374006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379373    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379451    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379521    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379519    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379576    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379655    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379695    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5b3rasg0scipm\""
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379738    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-llnck\""
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379760    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:06:38.379797 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379801    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:06:38.380324 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.379837    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:06:38.382742 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.382237    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:06:38.382742 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.382367    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:06:38.382742 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.382698    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:06:38.387020 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.387001    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:06:38.387319 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.387301    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:06:38.400560 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.400513    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:06:38.419003 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.418945    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerStarted","Data":"17dc037d9f19a96e1c66f1097ad74aa4062b5403bb9860cc39b1ae39ac82ac82"}
Oct 30 19:06:38.422817 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.422785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.423008 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.422827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423008 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.422904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.423008 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.422946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423008 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.422988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.423228 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423228 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423228 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423228 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423393 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.423393 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.423393 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423393 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423512 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjd94\" (UniqueName: \"kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.423544 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423630 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423630 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnw4\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423732 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423732 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423804 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423878 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423952 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.423987 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.424048 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.423993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.424124 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.424047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.424124 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.424110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.424237 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.424134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.424237 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.424200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.424618 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.424594    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps" event={"ID":"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634","Type":"ContainerStarted","Data":"4915fa67ab03bc4ef61d3cf83f604580ad2e25aa3ecf00f74c173065a05e62fb"}
Oct 30 19:06:38.424820 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.424800    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps" event={"ID":"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634","Type":"ContainerStarted","Data":"2a7683797e81b0e0ea214535effb2f08d0f7661a55abbcde786120f102680a19"}
Oct 30 19:06:38.424970 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.424957    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:38.425123 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.425101    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps" event={"ID":"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634","Type":"ContainerStarted","Data":"6f0164b20f63d00e9ca8bcd35690549744f062d83ec7fd4eb86456f9472f2259"}
Oct 30 19:06:38.426767 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.426736    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p" event={"ID":"77ff1252-ac70-4fad-a824-99f3c1eb860e","Type":"ContainerStarted","Data":"cf809d055ebd465e6ddc832e307eac06e6d7a25653f299b341f3b92b3e29cf9c"}
Oct 30 19:06:38.463803 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.463740    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.268991286 podStartE2EDuration="5.463720876s" totalImagesPullingTime="3.180885457s" totalInitContainerRuntime="13.844133ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:38.461645199 +0000 UTC m=+52.025108616" watchObservedRunningTime="2026-10-30 19:06:38.463720876 +0000 UTC m=+52.027184256"
Oct 30 19:06:38.489727 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.489672    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps" podStartSLOduration=1.7072678190000001 podStartE2EDuration="4.489656909s" totalImagesPullingTime="2.78238909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:38.486439763 +0000 UTC m=+52.049903142" watchObservedRunningTime="2026-10-30 19:06:38.489656909 +0000 UTC m=+52.053120286"
Oct 30 19:06:38.525572 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.525532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.525718 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.525602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.525718 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.525687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.525795 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.525751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.525795 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.525776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.525851 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.525819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.525881 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.525848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.525881 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.525872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.525956 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.525932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.526015 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.526045 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.526126 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.526184 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.526184 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.526279 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.526279 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.526382 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.526382 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.526482 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526573    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjd94\" (UniqueName: \"kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnw4\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.526944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.527001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.527243    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.527354    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.527570    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.528111 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.528021    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.531576 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.529269    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.531576 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.531400    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.532278 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.532255    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.533241 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.532728    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.533241 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.532750    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.533638 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.533617    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.535100 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.534786    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.535769 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.535290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.535769 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.535364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.535907 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.535822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.538278 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.536505    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.538278 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.536569    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.538278 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.537613    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.538278 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.537631    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.538278 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.538240    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.538554 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.538495    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.538943 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.538899    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.539248 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.539219    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.539386 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.539367    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.539441 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.539401    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.539949 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.539926    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.547198 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.547133    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnw4\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.547762 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.547741    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjd94\" (UniqueName: \"kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.668228 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.667947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:38.706190 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.705418    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:38.706190 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.705531    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:38.707313 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.706572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:38.711021 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.710974    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:38.960424 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:38.960393    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6474bf5b46-vc9zh"]
Oct 30 19:06:38.963684 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:38.963656    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61cc09d_5259_4d19_ab98_2cf90862bef2.slice/crio-1b797ba4614efd885de78c116712da8e7c1ea06d216d954dbf2978674be28bea WatchSource:0}: Error finding container 1b797ba4614efd885de78c116712da8e7c1ea06d216d954dbf2978674be28bea: Status 404 returned error can't find the container with id 1b797ba4614efd885de78c116712da8e7c1ea06d216d954dbf2978674be28bea
Oct 30 19:06:39.011041 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.011013    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:06:39.015158 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:39.015134    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d018801_0a52_45a6_a6c5_94692d11163a.slice/crio-9bbc37c289521525846e00af8ef2e919394bf5f1a73e51c791777243efefbbcc WatchSource:0}: Error finding container 9bbc37c289521525846e00af8ef2e919394bf5f1a73e51c791777243efefbbcc: Status 404 returned error can't find the container with id 9bbc37c289521525846e00af8ef2e919394bf5f1a73e51c791777243efefbbcc
Oct 30 19:06:39.431509 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.431468    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6474bf5b46-vc9zh" event={"ID":"a61cc09d-5259-4d19-ab98-2cf90862bef2","Type":"ContainerStarted","Data":"1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7"}
Oct 30 19:06:39.431957 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.431532    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6474bf5b46-vc9zh" event={"ID":"a61cc09d-5259-4d19-ab98-2cf90862bef2","Type":"ContainerStarted","Data":"1b797ba4614efd885de78c116712da8e7c1ea06d216d954dbf2978674be28bea"}
Oct 30 19:06:39.433026 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.433004    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-665cc87487-rzd5l" event={"ID":"f3925cae-4e22-41e0-b510-7c5cec2bfd17","Type":"ContainerStarted","Data":"33bc428e2d0fdf1717d8697269ec16b241f41ee80a9ba7ed345c22f2b8979f7b"}
Oct 30 19:06:39.434488 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.434463    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p" event={"ID":"77ff1252-ac70-4fad-a824-99f3c1eb860e","Type":"ContainerStarted","Data":"d259fc2422a84d6a77f904b2b64b3958bced40158a92d93e70c2742234289fe0"}
Oct 30 19:06:39.434648 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.434629    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:39.436126 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.436102    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d018801-0a52-45a6-a6c5-94692d11163a" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02" exitCode=0
Oct 30 19:06:39.436232 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.436211    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerDied","Data":"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"}
Oct 30 19:06:39.436284 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.436269    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerStarted","Data":"9bbc37c289521525846e00af8ef2e919394bf5f1a73e51c791777243efefbbcc"}
Oct 30 19:06:39.440017 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.439996    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:39.440484 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.440466    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:39.502013 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.501959    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6474bf5b46-vc9zh" podStartSLOduration=1.50194303 podStartE2EDuration="1.50194303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:39.459112561 +0000 UTC m=+53.022575940" watchObservedRunningTime="2026-10-30 19:06:39.50194303 +0000 UTC m=+53.065406404"
Oct 30 19:06:39.521291 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.521249    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-665cc87487-rzd5l" podStartSLOduration=1.781002709 podStartE2EDuration="3.521232478s" totalImagesPullingTime="1.740229769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:39.520374571 +0000 UTC m=+53.083837949" watchObservedRunningTime="2026-10-30 19:06:39.521232478 +0000 UTC m=+53.084696038"
Oct 30 19:06:39.542304 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:39.540549    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p" podStartSLOduration=2.080439805 podStartE2EDuration="3.54052498s" totalImagesPullingTime="1.460085175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:39.53666973 +0000 UTC m=+53.100133118" watchObservedRunningTime="2026-10-30 19:06:39.54052498 +0000 UTC m=+53.103988360"
Oct 30 19:06:41.301837 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:41.301804    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:42.372811 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:42.372785    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:42.453950 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:42.453829    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerStarted","Data":"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"}
Oct 30 19:06:42.453950 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:42.453917    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerStarted","Data":"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"}
Oct 30 19:06:43.461770 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:43.461727    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerStarted","Data":"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"}
Oct 30 19:06:43.462159 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:43.461814    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerStarted","Data":"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"}
Oct 30 19:06:43.462159 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:43.461854    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerStarted","Data":"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"}
Oct 30 19:06:43.462159 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:43.461893    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerStarted","Data":"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"}
Oct 30 19:06:43.499915 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:43.499857    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6434174219999997 podStartE2EDuration="5.499837653s" totalImagesPullingTime="2.839595339s" totalInitContainerRuntime="16.824892ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:43.496382467 +0000 UTC m=+57.059845845" watchObservedRunningTime="2026-10-30 19:06:43.499837653 +0000 UTC m=+57.063301035"
Oct 30 19:06:43.708767 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:43.708734    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:44.442171 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:44.442143    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:48.668927 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:48.668883    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:48.669329 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:48.668949    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:48.673376 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:48.673355    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:49.491204 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:49.491178    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:49.544713 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:49.544680    2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f645b69cc-d5dm6"]
Oct 30 19:06:52.569748 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.569697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:52.570162 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.569846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:52.571857 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.571834    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:06:52.571972 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.571959    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:06:52.582314 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.582286    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:52.582411 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.582356    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:52.760202 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.760171    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-89jvk\""
Oct 30 19:06:52.768525 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.768504    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:52.772044 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.772023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:52.774040 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.774026    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:06:52.778136 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.778117    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fljvz\""
Oct 30 19:06:52.784943 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.784898    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:06:52.787019 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.786999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:52.796203 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:52.796179    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:53.049682 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:53.049659    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tvwkt"]
Oct 30 19:06:53.052420 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:53.052383    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaec05dd_6d69_41d9_ad73_4179b17ba8f4.slice/crio-040dc6c51c24879b34281cc2599dd43229d4f81cf2677f2cb4a2cc31704cd2c4 WatchSource:0}: Error finding container 040dc6c51c24879b34281cc2599dd43229d4f81cf2677f2cb4a2cc31704cd2c4: Status 404 returned error can't find the container with id 040dc6c51c24879b34281cc2599dd43229d4f81cf2677f2cb4a2cc31704cd2c4
Oct 30 19:06:53.066272 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:53.066240    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tc7v2\""
Oct 30 19:06:53.067198 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:53.067146    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tbsh8"]
Oct 30 19:06:53.069142 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:53.069119    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceeebf2b_9b37_4988_b6c1_b8b41e09449a.slice/crio-8dda1c46d7fec2943c9aa718961354b77ff67f67f735523c811a3fd18fc030ac WatchSource:0}: Error finding container 8dda1c46d7fec2943c9aa718961354b77ff67f67f735523c811a3fd18fc030ac: Status 404 returned error can't find the container with id 8dda1c46d7fec2943c9aa718961354b77ff67f67f735523c811a3fd18fc030ac
Oct 30 19:06:53.074658 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:53.074641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:53.346613 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:53.346591    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k4plb"]
Oct 30 19:06:53.348606 ip-10-0-132-19 kubenswrapper[2578]: W1030 19:06:53.348572    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e38a783_1270_4703_a97c_cabdbe1c50ce.slice/crio-34a6f79cdcbe0acb2708ee56112521ab3ef871d2e97397ac9adfd73a9c3a1099 WatchSource:0}: Error finding container 34a6f79cdcbe0acb2708ee56112521ab3ef871d2e97397ac9adfd73a9c3a1099: Status 404 returned error can't find the container with id 34a6f79cdcbe0acb2708ee56112521ab3ef871d2e97397ac9adfd73a9c3a1099
Oct 30 19:06:53.505043 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:53.504997    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k4plb" event={"ID":"6e38a783-1270-4703-a97c-cabdbe1c50ce","Type":"ContainerStarted","Data":"34a6f79cdcbe0acb2708ee56112521ab3ef871d2e97397ac9adfd73a9c3a1099"}
Oct 30 19:06:53.507072 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:53.507027    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tbsh8" event={"ID":"ceeebf2b-9b37-4988-b6c1-b8b41e09449a","Type":"ContainerStarted","Data":"8dda1c46d7fec2943c9aa718961354b77ff67f67f735523c811a3fd18fc030ac"}
Oct 30 19:06:53.508599 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:53.508572    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tvwkt" event={"ID":"baec05dd-6d69-41d9-ad73-4179b17ba8f4","Type":"ContainerStarted","Data":"040dc6c51c24879b34281cc2599dd43229d4f81cf2677f2cb4a2cc31704cd2c4"}
Oct 30 19:06:54.518989 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:54.518952    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tbsh8" event={"ID":"ceeebf2b-9b37-4988-b6c1-b8b41e09449a","Type":"ContainerStarted","Data":"a425f4eef14b82efc44f0dba713eb0d08400dd97ee7c78b4b859cfc0523aceb8"}
Oct 30 19:06:55.525144 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:55.525075    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tbsh8" event={"ID":"ceeebf2b-9b37-4988-b6c1-b8b41e09449a","Type":"ContainerStarted","Data":"3e5e7928f0ae2021d6cb17a825631401b1af43ebf01f3272ff7de572955864b8"}
Oct 30 19:06:55.544749 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:55.544688    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tbsh8" podStartSLOduration=67.306835083 podStartE2EDuration="1m8.544670522s" totalImagesPullingTime="1.237835439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:55.542144812 +0000 UTC m=+69.105608190" watchObservedRunningTime="2026-10-30 19:06:55.544670522 +0000 UTC m=+69.108133899"
Oct 30 19:06:56.817446 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:56.816722    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:56.817446 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:56.816800    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:57.535173 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:57.535124    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k4plb" event={"ID":"6e38a783-1270-4703-a97c-cabdbe1c50ce","Type":"ContainerStarted","Data":"bdbbb1129bb23a8fb4cb7d7dbfa6b21186c233ab662b88fb5ab44554c35a67d9"}
Oct 30 19:06:57.535363 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:57.535245    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:57.563283 ip-10-0-132-19 kubenswrapper[2578]: I1030 19:06:57.563233    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k4plb" podStartSLOduration=67.414917741 podStartE2EDuration="1m10.563217676s" totalImagesPullingTime="3.148299935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:57.5615799 +0000 UTC m=+71.125043280" watchObservedRunningTime="2026-10-30 19:06:57.563217676 +0000 UTC m=+71.126681054"
Oct 30 19:06:57.946343 ip-10-0-132-19 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:06:57.956516 ip-10-0-132-19 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:06:57.956688 ip-10-0-132-19 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:06:57.956864 ip-10-0-132-19 systemd[1]: kubelet.service: Consumed 4.796s CPU time.
Oct 30 19:06:57.971894 ip-10-0-132-19 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:06:58.033479 ip-10-0-132-19 kubenswrapper[6777]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:06:58.033479 ip-10-0-132-19 kubenswrapper[6777]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:06:58.033479 ip-10-0-132-19 kubenswrapper[6777]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:06:58.033479 ip-10-0-132-19 kubenswrapper[6777]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:06:58.035503 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035417    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:58.035503 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035503    6777 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035509    6777 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035512    6777 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035515    6777 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035518    6777 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035521    6777 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035524    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035527    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035530    6777 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035534    6777 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035538    6777 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035541    6777 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035544    6777 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035550    6777 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035553    6777 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035556    6777 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035560    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035564    6777 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035568    6777 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:58.035580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035572    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035575    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035578    6777 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035581    6777 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035585    6777 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035588    6777 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035591    6777 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035593    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035596    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035599    6777 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035601    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035604    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035607    6777 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035609    6777 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035612    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035614    6777 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035617    6777 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035621    6777 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035623    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:58.036068 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035626    6777 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035629    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035631    6777 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035634    6777 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035637    6777 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035639    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035642    6777 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035645    6777 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035648    6777 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035651    6777 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035654    6777 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035656    6777 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035659    6777 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035662    6777 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035664    6777 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035667    6777 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035670    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035673    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035676    6777 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035678    6777 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:58.036547 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035681    6777 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035683    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035686    6777 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035689    6777 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035691    6777 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035694    6777 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035697    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035700    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035702    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035705    6777 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035707    6777 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035709    6777 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035712    6777 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035715    6777 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035717    6777 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035720    6777 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035722    6777 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035725    6777 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035728    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035731    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:58.037033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035733    6777 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035739    6777 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035742    6777 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035745    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035748    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035751    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035754    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035757    6777 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035760    6777 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035762    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035765    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035769    6777 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035771    6777 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035774    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035776    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035779    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035781    6777 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035784    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035787    6777 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:58.037530 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035789    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035792    6777 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035794    6777 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035797    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035799    6777 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035802    6777 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035804    6777 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.035807    6777 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036570    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036577    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036580    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036584    6777 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036587    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036589    6777 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036592    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036600    6777 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036603    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036606    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036608    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036611    6777 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:58.037987 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036614    6777 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036617    6777 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036619    6777 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036622    6777 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036625    6777 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036627    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036630    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036632    6777 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036635    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036637    6777 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036640    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036642    6777 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036645    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036647    6777 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036650    6777 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036653    6777 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036655    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036658    6777 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036661    6777 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:58.038500 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036663    6777 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036666    6777 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036668    6777 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036671    6777 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036673    6777 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036676    6777 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036678    6777 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036681    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036683    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036691    6777 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036694    6777 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036696    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036700    6777 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036702    6777 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036705    6777 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036707    6777 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036710    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036712    6777 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036715    6777 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036718    6777 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:58.038992 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036722    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036726    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036729    6777 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036734    6777 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036737    6777 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036740    6777 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036743    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036746    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036749    6777 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036751    6777 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036754    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036757    6777 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036759    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036762    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036765    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036767    6777 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036770    6777 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036772    6777 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:58.039524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036775    6777 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036777    6777 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036780    6777 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036782    6777 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036785    6777 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036788    6777 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036791    6777 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036793    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036796    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036798    6777 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036801    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036803    6777 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036806    6777 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036809    6777 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036811    6777 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036814    6777 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036816    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036819    6777 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036821    6777 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036824    6777 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:58.039975 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036830    6777 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036833    6777 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036835    6777 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036842    6777 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036845    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036847    6777 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036850    6777 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036852    6777 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036855    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036857    6777 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036860    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036862    6777 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036865    6777 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036867    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036870    6777 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036872    6777 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.036875    6777 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037357    6777 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037363    6777 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:58.040480 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037367    6777 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037371    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037374    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037377    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037380    6777 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037383    6777 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037385    6777 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037388    6777 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037390    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037393    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037396    6777 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037398    6777 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037401    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037403    6777 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037407    6777 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037409    6777 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037414    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037417    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037420    6777 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037423    6777 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:58.040972 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037425    6777 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037428    6777 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037430    6777 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037433    6777 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037436    6777 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037438    6777 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037441    6777 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037443    6777 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037445    6777 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037448    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037452    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037454    6777 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037457    6777 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037459    6777 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037462    6777 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037464    6777 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037467    6777 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037469    6777 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037471    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037474    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:58.041498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037477    6777 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037479    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037483    6777 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037486    6777 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037489    6777 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037492    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037494    6777 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037497    6777 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037501    6777 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037504    6777 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037506    6777 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037509    6777 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037511    6777 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037514    6777 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037516    6777 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037519    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037522    6777 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037524    6777 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037527    6777 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:58.042001 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037530    6777 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037532    6777 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037535    6777 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037537    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037540    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037543    6777 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037546    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037548    6777 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037551    6777 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037554    6777 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037556    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037558    6777 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037562    6777 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037565    6777 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037567    6777 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037569    6777 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037572    6777 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037574    6777 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037577    6777 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:58.042490 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037579    6777 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037582    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037586    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037589    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037591    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037594    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037596    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037599    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037602    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037605    6777 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037608    6777 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037611    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037614    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037616    6777 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037619    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037621    6777 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037624    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037626    6777 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037629    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:58.043008 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037632    6777 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037634    6777 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037637    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037639    6777 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037643    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037645    6777 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.037648    6777 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037737    6777 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037744    6777 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037750    6777 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037754    6777 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037758    6777 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037762    6777 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037766    6777 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037770    6777 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037775    6777 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037778    6777 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037782    6777 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037785    6777 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037788    6777 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037791    6777 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037794    6777 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037797    6777 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:06:58.043788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037800    6777 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037803    6777 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037807    6777 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037810    6777 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037813    6777 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037816    6777 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037820    6777 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037823    6777 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037827    6777 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037830    6777 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037834    6777 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037837    6777 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037840    6777 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037843    6777 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037846    6777 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037849    6777 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037853    6777 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037856    6777 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037859    6777 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037862    6777 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037865    6777 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037868    6777 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037872    6777 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037875    6777 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:06:58.044607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037880    6777 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037883    6777 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037886    6777 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037890    6777 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037893    6777 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037897    6777 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037900    6777 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037903    6777 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037905    6777 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037908    6777 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037911    6777 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037914    6777 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037917    6777 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037920    6777 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037924    6777 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037927    6777 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037931    6777 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037934    6777 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037937    6777 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037940    6777 flags.go:64] FLAG: --help="false"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037943    6777 flags.go:64] FLAG: --hostname-override="ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037946    6777 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037949    6777 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:06:58.045222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037952    6777 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037956    6777 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037959    6777 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037962    6777 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037964    6777 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037967    6777 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037970    6777 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037973    6777 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037976    6777 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037980    6777 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037984    6777 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037987    6777 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037990    6777 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037993    6777 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037996    6777 flags.go:64] FLAG: --lock-file=""
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.037999    6777 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038002    6777 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038006    6777 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038011    6777 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038014    6777 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038017    6777 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038020    6777 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038023    6777 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038027    6777 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:06:58.045782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038029    6777 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038032    6777 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038036    6777 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038040    6777 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038044    6777 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038048    6777 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038051    6777 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038066    6777 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038069    6777 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038072    6777 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038075    6777 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038079    6777 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038086    6777 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038089    6777 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038092    6777 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038095    6777 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038099    6777 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038103    6777 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038106    6777 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038109    6777 flags.go:64] FLAG: --port="10250"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038112    6777 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038115    6777 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0833cd20562798d9c"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038119    6777 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038121    6777 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038124    6777 flags.go:64] FLAG: --register-node="true"
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038127    6777 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:06:58.046411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038131    6777 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038134    6777 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038137    6777 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038158    6777 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038163    6777 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038166    6777 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038171    6777 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038174    6777 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038177    6777 flags.go:64] FLAG: --runonce="false"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038181    6777 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038184    6777 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038188    6777 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038191    6777 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038193    6777 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038196    6777 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038200    6777 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038203    6777 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038206    6777 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038209    6777 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038212    6777 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038215    6777 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038218    6777 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038221    6777 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038227    6777 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038233    6777 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:06:58.047071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038235    6777 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038238    6777 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038242    6777 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038245    6777 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038248    6777 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038251    6777 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038254    6777 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038257    6777 flags.go:64] FLAG: --v="2"
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038261    6777 flags.go:64] FLAG: --version="false"
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038265    6777 flags.go:64] FLAG: --vmodule=""
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038269    6777 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038272    6777 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038400    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038405    6777 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038409    6777 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038411    6777 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038414    6777 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038417    6777 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038419    6777 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038423    6777 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038425    6777 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038428    6777 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:58.047703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038430    6777 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038433    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038435    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038438    6777 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038440    6777 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038443    6777 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038446    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038448    6777 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038452    6777 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038456    6777 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038460    6777 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038463    6777 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038466    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038469    6777 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038472    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038474    6777 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038477    6777 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038479    6777 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038482    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:58.048341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038485    6777 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038488    6777 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038490    6777 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038493    6777 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038497    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038500    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038502    6777 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038505    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038508    6777 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038510    6777 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038513    6777 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038516    6777 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038519    6777 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038522    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038524    6777 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038526    6777 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038529    6777 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038531    6777 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038534    6777 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038537    6777 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:58.048795 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038539    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038542    6777 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038545    6777 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038547    6777 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038550    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038552    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038555    6777 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038557    6777 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038560    6777 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038562    6777 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038564    6777 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038567    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038570    6777 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038572    6777 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038575    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038577    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038581    6777 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038584    6777 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038586    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038589    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:58.049470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038591    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038594    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038596    6777 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038598    6777 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038601    6777 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038604    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038606    6777 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038609    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038611    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038613    6777 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038616    6777 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038618    6777 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038621    6777 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038623    6777 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038626    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038628    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038631    6777 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038633    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038636    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038638    6777 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:58.050088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038640    6777 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038643    6777 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038645    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038648    6777 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038650    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038653    6777 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038655    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038658    6777 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038662    6777 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038664    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038667    6777 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038669    6777 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038672    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038674    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038677    6777 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038679    6777 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:58.050696 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038683    6777 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.038709    6777 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038742    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038747    6777 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038750    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038753    6777 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038755    6777 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038758    6777 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038760    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038764    6777 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038766    6777 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038769    6777 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038771    6777 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038774    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038777    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038779    6777 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:58.051284 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038781    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038784    6777 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038786    6777 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038789    6777 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038791    6777 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038794    6777 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038796    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038799    6777 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038803    6777 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038805    6777 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038808    6777 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038811    6777 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038813    6777 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038816    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038818    6777 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038821    6777 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038823    6777 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038826    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038829    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038831    6777 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:58.051700 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038834    6777 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038836    6777 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038839    6777 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038841    6777 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038843    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038846    6777 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038848    6777 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038851    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038853    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038855    6777 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038858    6777 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038861    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038863    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038866    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038868    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038870    6777 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038873    6777 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038875    6777 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038880    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038882    6777 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:58.052391 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038886    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038889    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038891    6777 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038894    6777 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038896    6777 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038899    6777 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038901    6777 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038904    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038907    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038909    6777 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038912    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038915    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038918    6777 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038920    6777 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038930    6777 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038933    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038936    6777 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038938    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038941    6777 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:58.053017 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038944    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038946    6777 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038949    6777 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038952    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038954    6777 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038957    6777 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038978    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038981    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038984    6777 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038986    6777 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038990    6777 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.038996    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039000    6777 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039005    6777 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039009    6777 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039011    6777 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039014    6777 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039017    6777 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039020    6777 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:58.053557 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039022    6777 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039025    6777 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039027    6777 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039030    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039033    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039035    6777 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039038    6777 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039041    6777 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039044    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039046    6777 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039049    6777 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039053    6777 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039069    6777 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.039073    6777 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.044455    6777 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.044645    6777 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.044687    6777 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.044694    6777 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044815    6777 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:58.054069 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044829    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044834    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044839    6777 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044843    6777 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044848    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044853    6777 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044857    6777 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044861    6777 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044865    6777 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044870    6777 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044873    6777 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044876    6777 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044879    6777 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044882    6777 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044884    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044887    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044890    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044893    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044895    6777 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044898    6777 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:58.054536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044900    6777 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044903    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044906    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044909    6777 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044911    6777 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044914    6777 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044916    6777 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044920    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044923    6777 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044926    6777 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044928    6777 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044931    6777 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044934    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044938    6777 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044941    6777 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044945    6777 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044947    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044950    6777 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044953    6777 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:58.055028 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044955    6777 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044959    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044961    6777 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044964    6777 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044967    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044969    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044972    6777 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044974    6777 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044977    6777 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044979    6777 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044982    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044984    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044987    6777 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044990    6777 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044992    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044995    6777 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.044998    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045001    6777 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045003    6777 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045006    6777 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:58.055517 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045009    6777 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045012    6777 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045015    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045017    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045020    6777 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045023    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045026    6777 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045028    6777 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045032    6777 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045036    6777 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045039    6777 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045042    6777 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045045    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045049    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045052    6777 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045074    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045079    6777 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045082    6777 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045085    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:58.056003 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045089    6777 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045092    6777 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045095    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045097    6777 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045100    6777 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045102    6777 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045105    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045108    6777 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045110    6777 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045113    6777 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045115    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045118    6777 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045120    6777 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045124    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045126    6777 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045129    6777 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045132    6777 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045135    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045137    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045140    6777 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:58.056617 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045143    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045145    6777 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045148    6777 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045151    6777 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045154    6777 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045156    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045159    6777 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.045191    6777 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045337    6777 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045342    6777 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045345    6777 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045349    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045352    6777 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045355    6777 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045358    6777 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045360    6777 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:58.057115 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045363    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045366    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045369    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045371    6777 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045374    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045377    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045379    6777 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045383    6777 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045385    6777 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045389    6777 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045393    6777 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045396    6777 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045399    6777 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045402    6777 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045405    6777 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045408    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045411    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045414    6777 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045416    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:58.057576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045419    6777 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045421    6777 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045424    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045426    6777 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045429    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045432    6777 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045434    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045437    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045439    6777 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045442    6777 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045444    6777 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045447    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045450    6777 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045452    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045455    6777 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045458    6777 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045460    6777 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045463    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045465    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:58.058189 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045468    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045470    6777 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045473    6777 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045475    6777 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045478    6777 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045481    6777 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045484    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045486    6777 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045489    6777 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045491    6777 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045494    6777 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045496    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045499    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045502    6777 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045504    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045507    6777 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045510    6777 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045512    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045515    6777 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045517    6777 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:58.058646 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045520    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045522    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045525    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045527    6777 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045530    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045533    6777 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045535    6777 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045538    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045540    6777 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045542    6777 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045545    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045548    6777 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045551    6777 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045553    6777 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045556    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045559    6777 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045561    6777 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045563    6777 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045566    6777 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045568    6777 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:58.059267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045571    6777 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045573    6777 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045577    6777 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045581    6777 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045584    6777 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045586    6777 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045589    6777 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045592    6777 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045594    6777 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045597    6777 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045599    6777 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045602    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045604    6777 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045607    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045609    6777 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045612    6777 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045614    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045617    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045621    6777 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:58.059756 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045624    6777 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.045646    6777 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045676    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045680    6777 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045683    6777 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045686    6777 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045689    6777 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045691    6777 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045694    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045697    6777 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045700    6777 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045702    6777 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045705    6777 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045708    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045710    6777 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045713    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:58.060286 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045715    6777 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045718    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045721    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045723    6777 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045726    6777 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045729    6777 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045733    6777 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045737    6777 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045739    6777 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045742    6777 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045744    6777 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045747    6777 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045750    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045752    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045755    6777 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045757    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045760    6777 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045763    6777 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045766    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:58.060790 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045768    6777 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045771    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045773    6777 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045776    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045778    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045781    6777 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045784    6777 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045787    6777 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045793    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045796    6777 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045799    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045801    6777 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045804    6777 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045807    6777 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045809    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045812    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045815    6777 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045817    6777 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045820    6777 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045822    6777 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:58.061600 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045825    6777 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045827    6777 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045830    6777 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045832    6777 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045835    6777 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045837    6777 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045840    6777 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045842    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045845    6777 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045847    6777 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045851    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045854    6777 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045857    6777 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045859    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045861    6777 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045864    6777 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045866    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045869    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045871    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045874    6777 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:58.062357 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045878    6777 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045881    6777 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045884    6777 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045886    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045889    6777 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045891    6777 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045894    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045897    6777 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045899    6777 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045902    6777 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045904    6777 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045907    6777 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045909    6777 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045912    6777 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045914    6777 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045917    6777 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045919    6777 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045922    6777 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045925    6777 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045929    6777 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:58.062995 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045932    6777 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045935    6777 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045937    6777 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045940    6777 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045943    6777 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045945    6777 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045948    6777 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045950    6777 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045954    6777 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045957    6777 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045959    6777 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045962    6777 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:06:58.045967    6777 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.046045    6777 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.046998    6777 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.047076    6777 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.047261    6777 server.go:1014] "Starting client certificate rotation"
Oct 30 19:06:58.063706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.047395    6777 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:06:58.064344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.047449    6777 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:00:46 +0000 UTC" deadline="2028-04-02 12:17:20.402857604 +0000 UTC"
Oct 30 19:06:58.064344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.047477    6777 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12473h10m22.355382596s"
Oct 30 19:06:58.064344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.047751    6777 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:06:58.064344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.048867    6777 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:06:58.064344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.050589    6777 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:06:58.064344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.052693    6777 log.go:25] "Validated CRI v1 image API"
Oct 30 19:06:58.064344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.053448    6777 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:06:58.064344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.055819    6777 fs.go:135] Filesystem UUIDs: map[07aeb59a-2716-45de-b149-709b77101aa5:/dev/nvme0n1p3 408a1e8f-02d5-4175-be67-f3a77e4a2f89:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:06:58.064673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.055836    6777 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/040dc6c51c24879b34281cc2599dd43229d4f81cf2677f2cb4a2cc31704cd2c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/040dc6c51c24879b34281cc2599dd43229d4f81cf2677f2cb4a2cc31704cd2c4/userdata/shm major:0 minor:501 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07a73e09991a17edb14493abc70a24254ae29627a3b75ddbf6c0140d526aab4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07a73e09991a17edb14493abc70a24254ae29627a3b75ddbf6c0140d526aab4a/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1190742f738325737b9af75fe35985b2684552445fbda1a511d12a0882cb3f15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1190742f738325737b9af75fe35985b2684552445fbda1a511d12a0882cb3f15/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/157faf72a653c5be0d1f54bc2bc0d127aef12da8638b660a434e0ae5072b2b4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/157faf72a653c5be0d1f54bc2bc0d127aef12da8638b660a434e0ae5072b2b4b/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b797ba4614efd885de78c116712da8e7c1ea06d216d954dbf2978674be28bea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b797ba4614efd885de78c116712da8e7c1ea06d216d954dbf2978674be28bea/userdata/shm major:0 minor:442 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/207fd225e3156d80126edea5098e3657deb390fe9a23f6cbdd7322fd82d844d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/207fd225e3156d80126edea5098e3657deb390fe9a23f6cbdd7322fd82d844d3/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34a6f79cdcbe0acb2708ee56112521ab3ef871d2e97397ac9adfd73a9c3a1099/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34a6f79cdcbe0acb2708ee56112521ab3ef871d2e97397ac9adfd73a9c3a1099/userdata/shm major:0 minor:510 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/374a49a14c9b9712e1031be891f6677e3e5455bae39de6b4611ad49abca16d76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/374a49a14c9b9712e1031be891f6677e3e5455bae39de6b4611ad49abca16d76/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4764481df7a51d46ba985f6d991c7cb4ba0bc7a616154ce8f03af416e9b3d0ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4764481df7a51d46ba985f6d991c7cb4ba0bc7a616154ce8f03af416e9b3d0ba/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51f82c3dcd4f9f885f327fdfdd2cc76652dbe882b89beded02217427b1d72be6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51f82c3dcd4f9f885f327fdfdd2cc76652dbe882b89beded02217427b1d72be6/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57efde00068fa6c7bda32f3d95f526a5deee058013fe9d9755b61b1ad197587b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57efde00068fa6c7bda32f3d95f526a5deee058013fe9d9755b61b1ad197587b/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59f5ff61650f5215af4d4b0ae16aa6ddea87d2faabadaa6b06a368007b356490/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59f5ff61650f5215af4d4b0ae16aa6ddea87d2faabadaa6b06a368007b356490/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6136986741e80d17c0f61247a519e3761e3afb88427f9b341bf05ebef856b633/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6136986741e80d17c0f61247a519e3761e3afb88427f9b341bf05ebef856b633/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8389b33f6f123f5ffab77d0a7ac3f81ba9b7460ebc6892248ab3bdb028ae42eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8389b33f6f123f5ffab77d0a7ac3f81ba9b7460ebc6892248ab3bdb028ae42eb/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87dbbf18cd6875c46343477a1d5e975f5a46906ad5e7219ba14af2f603206796/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87dbbf18cd6875c46343477a1d5e975f5a46906ad5e7219ba14af2f603206796/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dda1c46d7fec2943c9aa718961354b77ff67f67f735523c811a3fd18fc030ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dda1c46d7fec2943c9aa718961354b77ff67f67f735523c811a3fd18fc030ac/userdata/shm major:0 minor:503 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91219e0e05ef48c11ad60ccf1335cbe7b1f267559355d90e6a84f839a4017459/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91219e0e05ef48c11ad60ccf1335cbe7b1f267559355d90e6a84f839a4017459/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91ec2d72013cd4defd209ccca364d3c6550f52a3e42234c7a8772b296a560ee0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91ec2d72013cd4defd209ccca364d3c6550f52a3e42234c7a8772b296a560ee0/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93ee2a35bd62881dac05f034f08a6d150907923921c956046f8846c0803ae693/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93ee2a35bd62881dac05f034f08a6d150907923921c956046f8846c0803ae693/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bbc37c289521525846e00af8ef2e919394bf5f1a73e51c791777243efefbbcc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bbc37c289521525846e00af8ef2e919394bf5f1a73e51c791777243efefbbcc/userdata/shm major:0 minor:444 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be123be39dc836e731c1d92f23e439dfa8e616a02caae81dc7ae272b875c0e94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be123be39dc836e731c1d92f23e439dfa8e616a02caae81dc7ae272b875c0e94/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8a85eb6caf3a654f8e227d9eb18c7c2e526ad1319967a86fd8aa37742188649/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8a85eb6caf3a654f8e227d9eb18c7c2e526ad1319967a86fd8aa37742188649/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf809d055ebd465e6ddc832e307eac06e6d7a25653f299b341f3b92b3e29cf9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf809d055ebd465e6ddc832e307eac06e6d7a25653f299b341f3b92b3e29cf9c/userdata/shm major:0 minor:403 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2525ac23364b8fad679e6fe5b360b73de2db27308552e63d46d72da7a3236fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2525ac23364b8fad679e6fe5b360b73de2db27308552e63d46d72da7a3236fa/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d761acb566f985e1207ae0b8637b8299da6f7f44b89cc59394a762c905c3c399/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d761acb566f985e1207ae0b8637b8299da6f7f44b89cc59394a762c905c3c399/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea79321c3382399cb3ef51cb5674f9dd0b1173c18bd1ee014e70008fdec4f4ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea79321c3382399cb3ef51cb5674f9dd0b1173c18bd1ee014e70008fdec4f4ea/userdata/shm major:0 minor:255 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/web-config/alertmanager/9 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/web-config/config-reloader/6 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~empty-dir/config-out major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~projected/kube-api-access-2xgsn:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~projected/kube-api-access-2xgsn major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~projected/tls-assets major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/config-volume major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/web-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b9c7068-7163-4c61-8a7f-5ec789247fa7/volumes/kubernetes.io~projected/kube-api-access-dw6wg:{mountpoint:/var/lib/kubelet/pods/0b9c7068-7163-4c61-8a7f-5ec789247fa7/volumes/kubernetes.io~projected/kube-api-access-dw6wg major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38ac8869-277b-44e4-a31a-bde13751efd9/volumes/kubernetes.io~projected/kube-api-access-g52v6:{mountpoint:/var/lib/kubelet/pods/38ac8869-277b-44e4-a31a-bde13751efd9/volumes/kubernetes.io~projected/kube-api-access-g52v6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46dd1c02-246a-41f2-bb6a-7523cc8b0436/volumes/kubernetes.io~projected/kube-api-access-csqhq:{mountpoint:/var/lib/kubelet/pods/46dd1c02-246a-41f2-bb6a-7523cc8b0436/volumes/kubernetes.io~projected/kube-api-access-csqhq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5232cfc3-47d5-46bb-8181-ecd05bc02348/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5232cfc3-47d5-46bb-8181-ecd05bc02348/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5232cfc3-47d5-46bb-8181-ecd05bc02348/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5232cfc3-47d5-46bb-8181-ecd05bc02348/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5232cfc3-47d5-46bb-8181-ecd05bc02348/volumes/kubernetes.io~projected/kube-api-access-mbk85:{mountpoint:/var/lib/kubelet/pods/5232cfc3-47d5-46bb-8181-ecd05bc02348/volumes/kubernetes.io~projected/kube-api-access-mbk85 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/558fbbeb-c828-4767-aecc-8c319ab91b35/volumes/kubernetes.io~projected/kube-api-access-5wdgm:{mountpoint:/var/lib/kubelet/pods/558fbbeb-c828-4767-aecc-8c319ab91b35/volumes/kubernetes.io~projected/kube-api-access-5wdgm major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/558fbbeb-c828-4767-aecc-8c319ab91b35/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/558fbbeb-c828-4767-aecc-8c319ab91b35/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/558fbbeb-c828-4767-aecc-8c319ab91b35/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/558fbbeb-c828-4767-aecc-8c319ab91b35/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60953151-cd1e-4572-a9f1-a618fee9ef83/volumes/kubernetes.io~projected/kube-api-access-qhvjr:{mountpoint:/var/lib/kubelet/pods/60953151-cd1e-4572-a9f1-a618fee9ef83/volumes/kubernetes.io~projected/kube-api-access-qhvjr major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60953151-cd1e-4572-a9f1-a618fee9ef83/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/60953151-cd1e-4572-a9f1-a618fee9ef83/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61317c8f-5a2e-4268-b085-df795ec4baa6/volumes/kubernetes.io~projected/kube-api-access-l8km5:{mountpoint:/var/lib/kubelet/pods/61317c8f-5a2e-4268-b085-df795ec4baa6/volumes/kubernetes.io~projected/kube-api-access-l8km5 major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61317c8f-5a2e-4268-b085-df795ec4baa6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/61317c8f-5a2e-4268-b085-df795ec4baa6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61317c8f-5a2e-4268-b085-df795ec4baa6/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/61317c8f-5a2e-4268-b085-df795ec4baa6/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e38a783-1270-4703-a97c-cabdbe1c50ce/volumes/kubernetes.io~projected/kube-api-access-b5px2:{mountpoint:/var/lib/kubelet/pods/6e38a783-1270-4703-a97c-cabdbe1c50ce/volumes/kubernetes.io~projected/kube-api-access-b5px2 major:0 minor:504 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77ff1252-ac70-4fad-a824-99f3c1eb860e/volumes/kubernetes.io~projected/kube-api-access-6r27l:{mountpoint:/var/lib/kubelet/pods/77ff1252-ac70-4fad-a824-99f3c1eb860e/volumes/kubernetes.io~projected/kube-api-access-6r27l major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77ff1252-ac70-4fad-a824-99f3c1eb860e/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/77ff1252-ac70-4fad-a824-99f3c1eb860e/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91513898-3858-4991-afad-cb5999ab98d0/volumes/kubernetes.io~projected/kube-api-access-hg9p5:{mountpoint:/var/lib/kubelet/pods/91513898-3858-4991-afad-cb5999ab98d0/volumes/kubernetes.io~projected/kube-api-access-hg9p5 major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91513898-3858-4991-afad-cb5999ab98d0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/91513898-3858-4991-afad-cb5999ab98d0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91513898-3858-4991-afad-cb5999ab98d0/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/91513898-3858-4991-afad-cb5999ab98d0/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volume-subpaths/web-config/config-reloader/2 major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volume-subpaths/web-config/prometheus/15 major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~projected/kube-api-access-2bnw4:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~projected/kube-api-access-2bnw4 major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~projected/tls-assets major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/config major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:419 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/web-config major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes/kubernetes.io~projected/kube-api-access-rjd94:{mountpoint:/var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes/kubernetes.io~projected/kube-api-access-rjd94 major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a979a593-d7fc-4659-a932-ca61a3801f27/volumes/kubernetes.io~projected/kube-api-access-frrl6:{mountpoint:/var/lib/kubelet/pods/a979a593-d7fc-4659-a932-ca61a3801f27/volumes/kubernetes.io~projected/kube-api-access-frrl6 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a979a593-d7fc-4659-a932-ca61a3801f27/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/a979a593-d7fc-4659-a932-ca61a3801f27/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3/volumes/kubernetes.io~projected/kube-api-access-b2nnm:{mountpoint:/var/lib/kubelet/pods/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3/volumes/kubernetes.io~projected/kube-api-access-b2nnm major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baec05dd-6d69-41d9-ad73-4179b17ba8f4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/baec05dd-6d69-41d9-ad73-4179b17ba8f4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:500 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf9f6e91-3cab-4b47-9526-a50b62527a16/volumes/kubernetes.io~projected/kube-api-access-9tmhg:{mountpoint:/var/lib/kubelet/pods/bf9f6e91-3cab-4b47-9526-a50b62527a16/volumes/kubernetes.io~projected/kube-api-access-9tmhg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb/volumes/kubernetes.io~projected/kube-api-access-mgsx5:{mountpoint:/var/lib/kubelet/pods/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb/volumes/kubernetes.io~projected/kube-api-access-mgsx5 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbb1844b-cbea-4ef6-a608-340b7ae7fee5/volumes/kubernetes.io~projected/kube-api-access-tp66z:{mountpoint:/var/lib/kubelet/pods/cbb1844b-cbea-4ef6-a608-340b7ae7fee5/volumes/kubernetes.io~projected/kube-api-access-tp66z major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ceeebf2b-9b37-4988-b6c1-b8b41e09449a/volumes/kubernetes.io~projected/kube-api-access-bmwz8:{mountpoint:/var/lib/kubelet/pods/ceeebf2b-9b37-4988-b6c1-b8b41e09449a/volumes/kubernetes.io~projected/kube-api-access-bmwz8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ceeebf2b-9b37-4988-b6c1-b8b41e09449a/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ceeebf2b-9b37-4988-b6c1-b8b41e09449a/volumes/kubernetes.io~secret/metrics-certs major:0 minor:496 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~projected/kube-api-access-fswgb:{mountpoint:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~projected/kube-api-access-fswgb major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes/kubernetes.io~projected/kube-api-access-xlrmg:{mountpoint:/var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes/kubernetes.io~projected/kube-api-access-xlrmg major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f31e88aa-14a5-4deb-a503-17b7f52cb4b9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f31e88aa-14a5-4deb-a503-17b7f52cb4b9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f31e88aa-14a5-4deb-a503-17b7f52cb4b9/volumes/kubernetes.io~projected/kube-api-access-z7l5r:{mountpoint:/var/lib/kubelet/pods/f31e88aa-14a5-4deb-a503-17b7f52cb4b9/volumes/kubernetes.io~projected/kube-api-access-z7l5r major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f31e88aa-14a5-4deb-a503-17b7f52cb4b9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f31e88aa-14a5-4deb-a503-17b7f52cb4b9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~projected/kube-api-access-l7vwl:{mountpoint:/var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~projected/kube-api-access-l7vwl major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:359 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/22e0d25c74e124135bdd708ff953d39f75cb72c4454de072bc215456c8e74085/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9fd9154c47195380a3bd52bb1f542df000b2ad0a388a836769920a93ad109941/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1ace1f2a3bf7ce830a0edd12b45447bd372e0fec82fee2bc5335e0b18027f030/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7af428892b6fe36de230f7adc3256c2260ddee46a073a143116899a02b9897cb/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3fd66b4d8b4bcd7dcbf36b5f54370996fe8ba2bfc4c5b4d915cededf97de3fab/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/49a13dbbee55e698bab99894fc1e1c9b661060cd8ae28b2afe1239b32619b900/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/825db0f56b13d388b05434955ba9f4b981da0408e7c84134270867bb5587cb35/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8d0126f89acd6c867f785d82cb9a2b857bbc2629d9b9b02f6f413d41f5671456/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/a3d502f82bae24a9836c22abc4554fe88cd06df5483386a225d8a464a5171252/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/412528a0f3365192c996ca7861bc0f3e540e7e02e892175b759cae6f1bc7c4a2/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/dc4d6ff0ae79c7e0ea5b16caa62ebabc1e8b36e90cfd8d916c607a72a3a39866/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/37f1cb25ba083e4e09b3c634c11140f8911d0ea3f0d715d9b76c13034623cb74/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/776ba1fdbd1d9c44ed534a4bdcf94a6c090ab6fc2df783f771e8fe034cb02a1f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1c9857d4a51db12727ea1957bada79781c4028879e24cf417cecd6105a0ada4b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/81ea0bb73229395161ea0ca8542f74c7b3411b2dd123ff0e7133b12b0afc5321/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/10582b5bfc08e53090c236b04463808e4c6952ebc51db9439da9be8a0e2346d7/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ebaf9e5f3280adda48d8a1abf42a03b3ce002281a846ea18c3b92ae7b2652c48/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/49e0dac6fc6bea65b131e2fc6d36c07689f15e2d7efc965943f72fb93db389ad/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ab971db086296f978f43c7f0b39d3aa299123817bbba158335f5eff95a5cb2e9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/874bb461713b2e859e239181bf48594cad9a720cde40bed98ba13f3116822452/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/989eabcfe6eef215a708f860c1deae7c3b3d82ae4b0c7a7306a4380369d25e22/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/7f47432776603a9b41798d456f37598d7d15c64c213051f89f1a1dd593cefc77/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/7a5c0a576afc3c5a1bb899a72e48dadce5021a0de978389f1320df9efccf57ca/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/057ec5e454b20262aca7a7a4cbaac7cd873f1b6a6c3c6d0edda64dfcecb8efc0/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/0a44f3785bca85d03c9929ab0ab1ebad26fd7bb958cec395a765ce74cf132a76/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/4b6362d7ee341ce82b2a8e47a5e47033f79c93f37c35c296e5fd9d415da4c6b5/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/c02ca0bf893f261783cb7fe8e57b26fd0864de3e550118119343c9b46be6ef7b/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/8f95a7c4e213a8277230fb28b0382407864c855636251de7a10aae30eb832492/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/c2f5bdd9dda6b2883e0f1641c8de0034894f666e64c71bc87ac22f64e93f4c91/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/4432c911837390383f6cc27afec93fdf64bd82cfb6c82779ee7754bda1acde84/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/5d007aea4efc1c7297e78ce3a7bb6ac56e8477747933ba33ccbbad3e469bc2b7/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/925b10b6cf966a08afe5b3d24ae751836c7b1e609759368f7b6593f7e237a382/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/3d0b07a5036482bbfa641173f47f17da543711be7ee854280a3f7d9abc48aa5d/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/40566a268723b22b0b29deded3cc4ae7713af20e5c6b118d1be8a914c8a147c9/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/84a6e04d33cccaad81cb2c583ec9b1b8d2720cad67fffc5791eb5dbecc5d6728/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/a4673463c716be39c47ffb9eacd34652551e3c510f30f17c79014f610547d9b1/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/2bf90400e4adf91adfae89f81390070cfd8d5b2dddf09bb1cc263fe7587d28c8/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/090274f2bb85ebf68605ca4806691616f7d631685bec223cbaae659654c89613/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/b1338a87628f32b0c9f86862d21ed31998853dcf1fc06d088db0ae67534681e7/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/82c4d5ca6222b82e5b0bafd0528b69fdc5c3a006b76dd23544dadc270bda81c6/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/592f7f198ca9a4d1b655444f6cf5c6b2fc5526a93e7b7dbbf04b5e7687df08c8/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/6350456a969814e36e719dfaff9e571c9b02cf3fec4aac7b7a84f9bc5419fae2/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/477a7bff6fe4c63ed8e977ef95d2ecabee520837ed4995c6dc80ad5493c8ac74/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/36cfcc2f351006321dda2efe6652cc6a944964546c50baec90bf4c8271fa11ce/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/15cacdede88c3710cdea89063fd798042a4ed45a2f64a17ea116b120a7f6219c/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/794ba4d9c46fbddcb6f5f3d658e99ac27322401f1dd666ed0dd93e346544968a/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/03d75591fb30fc1695447bbffa4e0623ff6e6fb05736d9a2fc3a477b81e44fd1/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/aab3e2639d867f284dc76b35ad401c2d80d5b63feb469b7d861a3c840d2d4b61/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/e9e61dfbe87e84ca43f90b4d486a573029ebcd9df3b6e83e4eb3383bad8b4c26/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/1e839f7134d7ac3b7a9ef580e892a29bb03d22d6a0516ed8e15e60fbaa6c823b/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/4dc4833835df6f53c06329c42393602db19c3aea7309caa346b44d94e090c8ec/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/40387aa83c8f7d0f615e8f2dd84d286974f12d54cae53a4df6273b9434673f10/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/1cc06e845b10ee6fc626401f2598407d4011092eefd07ad66dfca6561c56e056/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/698ece33ec909828017f7d5fbfa9534cff7b425afa069fa7c78e2d269d245784/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/2fdcb13db3bdacc6990caeda4de8e6cc79055ace6d04e36224a25ccfee2d542a/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/1ab23b39dbfe3d6680d0a4db981c9a63c409d336fd67067d5318525e5eb3bd15/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/b21602e43399b5e3395106cb611e24171f6c2a7618801f36ce0ba72713a55250/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/e9c3a6989f0790c6cb58c9d49826d8ffda60b014f172813c4aeee4fe1dbed2f9/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/cf43d9431eed57366b8e8e636a9e9f12b2fb8744f526a00bf74a90182fb141dd/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/337a90261b91568d1726ece62099982c398edd83cbfba7ae80dc050698812dbd/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/d4312cff6646be09f1531c5f99e6b97c5e22cbbae8f6554ff13fecc38188b5e1/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/9435df957e05506f7aa5129262a14e5efb53d8011d269eea3897eb44fd42f3ec/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/1fc131a3acf8d0ee34e805f4e8eae5f5dd09842d5161cb02e370760dbfbb6a6f/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/040392f4b94d30ddab4c3f1e4632f248275bedc09631564dadc702fc873755ab/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/a02a058d6ff1d0e43b7f1c151c397c7ad5f5156a4810ca39758455371e4fdd50/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-470:{mountpoint:/var/lib/containers/storage/overlay/f33f222b299ae032d1c39976b07a2cac0d3fea483d627a787344a30b765225ad/merged major:0 minor:470 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/d517d53a62b0b907d0c3776d807488e661908a689f64709d9e7ec3361e30c175/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/c86a6b60a6fa607dd8fff6d12408b7b79da017dd9890c04a6f532e7faebe08da/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/5fd7303579ce0b30b92468f3aff2ace560bd07a754a64c7861a5d6c7b4ca4c3f/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/f383dea04005ab42cd9c601c6a4782c0d109778a8a9362dd1b54b80539a57e31/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/ce00d356db1aa39f20e23eca5b2ec35386902243aee0abbf50bab32c5929e28a/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-508:{mountpoint:/var/lib/containers/storage/overlay/bc2eed2a66c3eb214a8ec59dd01318f3d0ca4ab6ca564b1745421230e625491e/merged major:0 minor:508 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/4eacdea8157976c3064c589c1d9daff4cdba11b1dee9ad6f7d53c7ecf9983f16/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-512:{mountpoint:/var/lib/containers/storage/overlay/d6c21eb7a61780203ed466861458ba99c01ed4d9cce95286d779a507a4190f47/merged major:0 minor:512 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/92699918ac1108a5009ddc30b24123f5417491c0dc0351d4ae9bd4541141f3ee/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-516:{mountpoint:/var/lib/containers/storage/overlay/5681e20ac4e655ff12bf1917450c6a150366f2547f6cf517840bbbf5c208a511/merged major:0 minor:516 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/8fee9a1a1b56d0dc09ab522850257ad3c9d348b76605dd68cfea90bbca65eb3d/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-527:{mountpoint:/var/lib/containers/storage/overlay/575648cb5f8c6b7c2324f83f46e1ed424260b96bd3cdee4c681c1f5ac5f2f115/merged major:0 minor:527 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8cd19422fe9e6c14f1ca0bfea207a67e95666313b3bd101063c3a7ea8f3ee3e6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/ad2e789d6d9a55d57f5325ce3f682660e03db7295e2b9b1eb093eeb7b914aad7/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/601c3b1b2479ed7a8c8105fc69881be936078556c799d51f42a5e3d095f71289/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/85546687c27f870c4f2f87b254108c627ce54019b0074b51655175a788bd7c81/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/603e1049920819ef2f81cf5310f0f2a727c883633a3ac976b835947281b8803c/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/5d280afe78eee51477b6c7bf8d9222ed006a963852ca9590a89e3b614405c560/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/16fa84fcebd00f8cbb2e2281167d99ebedd9c0cd699042e8a8fa6fc7948fbd88/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c283fa39ab0f691bc8a83d42cb4499a2f305a3a213a259f8af6792b7e8d40131/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7fc76b3ded5c563411bdbe2eee806ae7a7240613b62a4b02cc49ec48241f2b98/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:06:58.066161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.065647    6777 manager.go:217] Machine: {Timestamp:2026-10-30 19:06:58.065193863 +0000 UTC m=+0.062109245 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089138 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d129208b79bad133d6d7d185a1516 SystemUUID:ec2d1292-08b7-9bad-133d-6d7d185a1516 BootID:408b0993-0a4f-4567-8c6f-b83d1d0420b8 Filesystems:[{Device:/run/containers/storage/overlay-containers/1b797ba4614efd885de78c116712da8e7c1ea06d216d954dbf2978674be28bea/userdata/shm DeviceMajor:0 DeviceMinor:442 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:433 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:439 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:432 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:437 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a979a593-d7fc-4659-a932-ca61a3801f27/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:437 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/61317c8f-5a2e-4268-b085-df795ec4baa6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:249 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91513898-3858-4991-afad-cb5999ab98d0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:434 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:437 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a979a593-d7fc-4659-a932-ca61a3801f27/volumes/kubernetes.io~projected/kube-api-access-frrl6 DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59f5ff61650f5215af4d4b0ae16aa6ddea87d2faabadaa6b06a368007b356490/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:425 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bbc37c289521525846e00af8ef2e919394bf5f1a73e51c791777243efefbbcc/userdata/shm DeviceMajor:0 DeviceMinor:444 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/040dc6c51c24879b34281cc2599dd43229d4f81cf2677f2cb4a2cc31704cd2c4/userdata/shm DeviceMajor:0 DeviceMinor:501 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f31e88aa-14a5-4deb-a503-17b7f52cb4b9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~projected/kube-api-access-2xgsn DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/baec05dd-6d69-41d9-ad73-4179b17ba8f4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:500 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ceeebf2b-9b37-4988-b6c1-b8b41e09449a/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:496 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:317 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~projected/kube-api-access-2bnw4 DeviceMajor:0 DeviceMinor:440 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be123be39dc836e731c1d92f23e439dfa8e616a02caae81dc7ae272b875c0e94/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60953151-cd1e-4572-a9f1-a618fee9ef83/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60953151-cd1e-4572-a9f1-a618fee9ef83/volumes/kubernetes.io~projected/kube-api-access-qhvjr DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:292 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf9f6e91-3cab-4b47-9526-a50b62527a16/volumes/kubernetes.io~projected/kube-api-access-9tmhg DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57efde00068fa6c7bda32f3d95f526a5deee058013fe9d9755b61b1ad197587b/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes/kubernetes.io~projected/kube-api-access-xlrmg DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~projected/kube-api-access-fswgb DeviceMajor:0 DeviceMinor:320 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf809d055ebd465e6ddc832e307eac06e6d7a25653f299b341f3b92b3e29cf9c/userdata/shm DeviceMajor:0 DeviceMinor:403 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2525ac23364b8fad679e6fe5b360b73de2db27308552e63d46d72da7a3236fa/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d761acb566f985e1207ae0b8637b8299da6f7f44b89cc59394a762c905c3c399/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91513898-3858-4991-afad-cb5999ab98d0/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5232cfc3-47d5-46bb-8181-ecd05bc02348/volumes/kubernetes.io~projected/kube-api-access-mbk85 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f31e88aa-14a5-4deb-a503-17b7f52cb4b9/volumes/kubernetes.io~projected/kube-api-access-z7l5r DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07a73e09991a17edb14493abc70a24254ae29627a3b75ddbf6c0140d526aab4a/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:429 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:437 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51f82c3dcd4f9f885f327fdfdd2cc76652dbe882b89beded02217427b1d72be6/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/558fbbeb-c828-4767-aecc-8c319ab91b35/volumes/kubernetes.io~projected/kube-api-access-5wdgm DeviceMajor:0 DeviceMinor:250 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~projected/kube-api-access-l7vwl DeviceMajor:0 DeviceMinor:362 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3/volumes/kubernetes.io~projected/kube-api-access-b2nnm DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3925cae-4e22-41e0-b510-7c5cec2bfd17/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:359 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/77ff1252-ac70-4fad-a824-99f3c1eb860e/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:390 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/77ff1252-ac70-4fad-a824-99f3c1eb860e/volumes/kubernetes.io~projected/kube-api-access-6r27l DeviceMajor:0 DeviceMinor:386 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/61317c8f-5a2e-4268-b085-df795ec4baa6/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:243 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4764481df7a51d46ba985f6d991c7cb4ba0bc7a616154ce8f03af416e9b3d0ba/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91219e0e05ef48c11ad60ccf1335cbe7b1f267559355d90e6a84f839a4017459/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/374a49a14c9b9712e1031be891f6677e3e5455bae39de6b4611ad49abca16d76/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5232cfc3-47d5-46bb-8181-ecd05bc02348/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87dbbf18cd6875c46343477a1d5e975f5a46906ad5e7219ba14af2f603206796/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/558fbbeb-c828-4767-aecc-8c319ab91b35/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb/volumes/kubernetes.io~projected/kube-api-access-mgsx5 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6136986741e80d17c0f61247a519e3761e3afb88427f9b341bf05ebef856b633/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes/kubernetes.io~projected/kube-api-access-rjd94 DeviceMajor:0 DeviceMinor:441 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-512 DeviceMajor:0 DeviceMinor:512 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:430 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-470 DeviceMajor:0 DeviceMinor:470 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/157faf72a653c5be0d1f54bc2bc0d127aef12da8638b660a434e0ae5072b2b4b/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dda1c46d7fec2943c9aa718961354b77ff67f67f735523c811a3fd18fc030ac/userdata/shm DeviceMajor:0 DeviceMinor:503 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91513898-3858-4991-afad-cb5999ab98d0/volumes/kubernetes.io~projected/kube-api-access-hg9p5 DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1190742f738325737b9af75fe35985b2684552445fbda1a511d12a0882cb3f15/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:431 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38ac8869-277b-44e4-a31a-bde13751efd9/volumes/kubernetes.io~projected/kube-api-access-g52v6 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e38a783-1270-4703-a97c-cabdbe1c50ce/volumes/kubernetes.io~projected/kube-api-access-b5px2 DeviceMajor:0 DeviceMinor:504 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8389b33f6f123f5ffab77d0a7ac3f81ba9b7460ebc6892248ab3bdb028ae42eb/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5232cfc3-47d5-46bb-8181-ecd05bc02348/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbb1844b-cbea-4ef6-a608-340b7ae7fee5/volumes/kubernetes.io~projected/kube-api-access-tp66z DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/207fd225e3156d80126edea5098e3657deb390fe9a23f6cbdd7322fd82d844d3/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/61317c8f-5a2e-4268-b085-df795ec4baa6/volumes/kubernetes.io~projected/kube-api-access-l8km5 DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-527 DeviceMajor:0 DeviceMinor:527 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91ec2d72013cd4defd209ccca364d3c6550f52a3e42234c7a8772b296a560ee0/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34a6f79cdcbe0acb2708ee56112521ab3ef871d2e97397ac9adfd73a9c3a1099/userdata/shm DeviceMajor:0 DeviceMinor:510 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-516 DeviceMajor:0 DeviceMinor:516 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f31e88aa-14a5-4deb-a503-17b7f52cb4b9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93ee2a35bd62881dac05f034f08a6d150907923921c956046f8846c0803ae693/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:419 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ceeebf2b-9b37-4988-b6c1-b8b41e09449a/volumes/kubernetes.io~projected/kube-api-access-bmwz8 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8a85eb6caf3a654f8e227d9eb18c7c2e526ad1319967a86fd8aa37742188649/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b9c7068-7163-4c61-8a7f-5ec789247fa7/volumes/kubernetes.io~projected/kube-api-access-dw6wg DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/558fbbeb-c828-4767-aecc-8c319ab91b35/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:248 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:318 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:438 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-508 DeviceMajor:0 DeviceMinor:508 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46dd1c02-246a-41f2-bb6a-7523cc8b0436/volumes/kubernetes.io~projected/kube-api-access-csqhq DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea79321c3382399cb3ef51cb5674f9dd0b1173c18bd1ee014e70008fdec4f4ea/userdata/shm DeviceMajor:0 DeviceMinor:255 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:435 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:040dc6c51c24879 MacAddress:4e:7c:ae:20:24:37 Speed:10000 Mtu:8901} {Name:1190742f7383257 MacAddress:32:28:f2:f3:6c:a1 Speed:10000 Mtu:8901} {Name:157faf72a653c5b MacAddress:7a:d4:37:3a:dc:ca Speed:10000 Mtu:8901} {Name:1b797ba4614efd8 MacAddress:0e:56:48:3b:4a:ea Speed:10000 Mtu:8901} {Name:207fd225e3156d8 MacAddress:c6:30:04:08:aa:ad Speed:10000 Mtu:8901} {Name:34a6f79cdcbe0ac MacAddress:9e:67:c2:5f:29:53 Speed:10000 Mtu:8901} {Name:6136986741e80d1 MacAddress:fe:f3:42:9a:f7:37 Speed:10000 Mtu:8901} {Name:8dda1c46d7fec29 MacAddress:ba:64:a8:32:5b:5f Speed:10000 Mtu:8901} {Name:91ec2d72013cd4d MacAddress:12:09:3e:75:32:e5 Speed:10000 Mtu:8901} {Name:93ee2a35bd62881 MacAddress:f6:ee:96:9f:94:0c Speed:10000 Mtu:8901} {Name:9bbc37c28952152 MacAddress:2e:6c:7e:f8:a2:f4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e5:90:8b:db Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:45:73:ba:40:c4 Speed:0 Mtu:8901} {Name:c8a85eb6caf3a65 MacAddress:da:a6:ec:62:ad:7a Speed:10000 Mtu:8901} {Name:cf809d055ebd465 MacAddress:ea:2a:4f:1e:41:1a Speed:10000 Mtu:8901} {Name:d761acb566f985e MacAddress:b2:23:95:b6:91:bc Speed:10000 Mtu:8901} {Name:ea79321c3382399 MacAddress:e6:9f:9d:c6:dd:fc Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e5:90:8b:db Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:82:8d:87:75:c0:dc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:b9:34:36:fe:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:06:58.066482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066166    6777 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:06:58.066482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066214    6777 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:06:58.066545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066502    6777 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:06:58.066648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066526    6777 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-19.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:06:58.066695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066650    6777 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:06:58.066695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066658    6777 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:06:58.066695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066671    6777 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:06:58.066695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066684    6777 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:06:58.066794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066768    6777 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:06:58.066922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.066912    6777 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:06:58.067069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.067047    6777 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:06:58.067122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.067105    6777 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:06:58.067122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.067118    6777 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:06:58.067187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.067129    6777 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:06:58.067187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.067137    6777 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:06:58.067187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.067145    6777 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:06:58.068274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.068254    6777 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:06:58.068274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.068271    6777 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:06:58.068871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.068851    6777 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:06:58.072366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072349    6777 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:06:58.072366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072362    6777 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:06:58.072475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072450    6777 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:06:58.072639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072627    6777 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:06:58.072673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072656    6777 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:06:58.072746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072737    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:06:58.072781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072751    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:06:58.072781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072759    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:06:58.072781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072764    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:06:58.072781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072770    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:06:58.072781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072775    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:06:58.072950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072785    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:06:58.072950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072791    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:06:58.072950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072797    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:06:58.072950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072803    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:06:58.072950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072812    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:06:58.072950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072822    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:06:58.072950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072852    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:06:58.072950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.072858    6777 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:06:58.076245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.076231    6777 server.go:1263] "Started kubelet"
Oct 30 19:06:58.076333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.076300    6777 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:06:58.076404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.076316    6777 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:06:58.076451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.076428    6777 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:06:58.076783 ip-10-0-132-19 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:06:58.077551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.077535    6777 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:06:58.078362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.078339    6777 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:06:58.082616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.082597    6777 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:06:58.082616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.082611    6777 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:06:58.085471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.082665    6777 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:00:48 +0000 UTC" deadline="2028-04-28 18:18:29.672258258 +0000 UTC"
Oct 30 19:06:58.085471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.082696    6777 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13103h11m31.589565617s"
Oct 30 19:06:58.085471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.082705    6777 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:06:58.085471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.082717    6777 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:06:58.085471 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:06:58.082724    6777 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-19.us-east-2.compute.internal\" not found"
Oct 30 19:06:58.085471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.082932    6777 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:06:58.086293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086270    6777 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:06:58.086390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086295    6777 factory.go:55] Registering systemd factory
Oct 30 19:06:58.086390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086308    6777 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:06:58.086587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086525    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46dd1c02-246a-41f2-bb6a-7523cc8b0436" volumeName="kubernetes.io/projected/46dd1c02-246a-41f2-bb6a-7523cc8b0436-kube-api-access-csqhq" seLinuxMountContext=""
Oct 30 19:06:58.086633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086593    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="558fbbeb-c828-4767-aecc-8c319ab91b35" volumeName="kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:06:58.086633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086612    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38ac8869-277b-44e4-a31a-bde13751efd9" volumeName="kubernetes.io/projected/38ac8869-277b-44e4-a31a-bde13751efd9-kube-api-access-g52v6" seLinuxMountContext=""
Oct 30 19:06:58.086633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086626    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls" seLinuxMountContext=""
Oct 30 19:06:58.086722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086640    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91513898-3858-4991-afad-cb5999ab98d0" volumeName="kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:06:58.086722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086653    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61cc09d-5259-4d19-ab98-2cf90862bef2" volumeName="kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94" seLinuxMountContext=""
Oct 30 19:06:58.086722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086665    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9f6e91-3cab-4b47-9526-a50b62527a16" volumeName="kubernetes.io/projected/bf9f6e91-3cab-4b47-9526-a50b62527a16-kube-api-access-9tmhg" seLinuxMountContext=""
Oct 30 19:06:58.086722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086681    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" volumeName="kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:06:58.086722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086696    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 19:06:58.086722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086710    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baec05dd-6d69-41d9-ad73-4179b17ba8f4" volumeName="kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086718    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60953151-cd1e-4572-a9f1-a618fee9ef83" volumeName="kubernetes.io/empty-dir/60953151-cd1e-4572-a9f1-a618fee9ef83-data-volume" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086736    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086745    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceeebf2b-9b37-4988-b6c1-b8b41e09449a" volumeName="kubernetes.io/projected/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-kube-api-access-bmwz8" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086752    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60953151-cd1e-4572-a9f1-a618fee9ef83" volumeName="kubernetes.io/projected/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-api-access-qhvjr" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086762    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e38a783-1270-4703-a97c-cabdbe1c50ce" volumeName="kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086769    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61cc09d-5259-4d19-ab98-2cf90862bef2" volumeName="kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086778    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634" volumeName="kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086786    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086793    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634" volumeName="kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086801    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634" volumeName="kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086809    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086816    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38ac8869-277b-44e4-a31a-bde13751efd9" volumeName="kubernetes.io/empty-dir/38ac8869-277b-44e4-a31a-bde13751efd9-tmp-dir" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086823    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60953151-cd1e-4572-a9f1-a618fee9ef83" volumeName="kubernetes.io/configmap/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086833    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61317c8f-5a2e-4268-b085-df795ec4baa6" volumeName="kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086840    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086847    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086854    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a979a593-d7fc-4659-a932-ca61a3801f27" volumeName="kubernetes.io/configmap/a979a593-d7fc-4659-a932-ca61a3801f27-config-volume" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086861    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaa77658-47ce-47ee-b0a3-ec9dc492c3b3" volumeName="kubernetes.io/secret/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-cert" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086871    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config" seLinuxMountContext=""
Oct 30 19:06:58.086875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086879    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61317c8f-5a2e-4268-b085-df795ec4baa6" volumeName="kubernetes.io/configmap/61317c8f-5a2e-4268-b085-df795ec4baa6-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086886    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086903    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086914    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086931    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61cc09d-5259-4d19-ab98-2cf90862bef2" volumeName="kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086939    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a979a593-d7fc-4659-a932-ca61a3801f27" volumeName="kubernetes.io/projected/a979a593-d7fc-4659-a932-ca61a3801f27-kube-api-access-frrl6" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086975    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a979a593-d7fc-4659-a932-ca61a3801f27" volumeName="kubernetes.io/secret/a979a593-d7fc-4659-a932-ca61a3801f27-metrics-tls" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.086993    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9f6e91-3cab-4b47-9526-a50b62527a16" volumeName="kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087002    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" volumeName="kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087009    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087017    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="558fbbeb-c828-4767-aecc-8c319ab91b35" volumeName="kubernetes.io/empty-dir/558fbbeb-c828-4767-aecc-8c319ab91b35-volume-directive-shadow" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087030    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91513898-3858-4991-afad-cb5999ab98d0" volumeName="kubernetes.io/empty-dir/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-textfile" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087038    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91513898-3858-4991-afad-cb5999ab98d0" volumeName="kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-tls" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087045    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" volumeName="kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087052    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31e88aa-14a5-4deb-a503-17b7f52cb4b9" volumeName="kubernetes.io/projected/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-kube-api-access-z7l5r" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087145    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5232cfc3-47d5-46bb-8181-ecd05bc02348" volumeName="kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-tuned" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087164    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="558fbbeb-c828-4767-aecc-8c319ab91b35" volumeName="kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087177    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634" volumeName="kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-grpc-tls" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087186    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3925cae-4e22-41e0-b510-7c5cec2bfd17" volumeName="kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087195    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b9c7068-7163-4c61-8a7f-5ec789247fa7" volumeName="kubernetes.io/projected/0b9c7068-7163-4c61-8a7f-5ec789247fa7-kube-api-access-dw6wg" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087205    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b26e19e-b8e9-496a-bfcb-6cd5add61e6e" volumeName="kubernetes.io/configmap/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087212    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087220    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087228    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087238    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebd094-601c-462b-a42c-89e9d6a27c2a" volumeName="kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087245    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebd094-601c-462b-a42c-89e9d6a27c2a" volumeName="kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087252    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebd094-601c-462b-a42c-89e9d6a27c2a" volumeName="kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087273    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3925cae-4e22-41e0-b510-7c5cec2bfd17" volumeName="kubernetes.io/projected/f3925cae-4e22-41e0-b510-7c5cec2bfd17-kube-api-access-l7vwl" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087281    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77ff1252-ac70-4fad-a824-99f3c1eb860e" volumeName="kubernetes.io/projected/77ff1252-ac70-4fad-a824-99f3c1eb860e-kube-api-access-6r27l" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087292    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087299    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="558fbbeb-c828-4767-aecc-8c319ab91b35" volumeName="kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087307    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77ff1252-ac70-4fad-a824-99f3c1eb860e" volumeName="kubernetes.io/secret/77ff1252-ac70-4fad-a824-99f3c1eb860e-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087313    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087321    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61cc09d-5259-4d19-ab98-2cf90862bef2" volumeName="kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087341    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebd094-601c-462b-a42c-89e9d6a27c2a" volumeName="kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087355    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebd094-601c-462b-a42c-89e9d6a27c2a" volumeName="kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087370    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3925cae-4e22-41e0-b510-7c5cec2bfd17" volumeName="kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087384    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087391    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61317c8f-5a2e-4268-b085-df795ec4baa6" volumeName="kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087407    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087415    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634" volumeName="kubernetes.io/empty-dir/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-tmp" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087421    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3925cae-4e22-41e0-b510-7c5cec2bfd17" volumeName="kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087436    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b26e19e-b8e9-496a-bfcb-6cd5add61e6e" volumeName="kubernetes.io/secret/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-agent-certs" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087466    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5232cfc3-47d5-46bb-8181-ecd05bc02348" volumeName="kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-tmp" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087479    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceeebf2b-9b37-4988-b6c1-b8b41e09449a" volumeName="kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087508    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087520    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087531    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61cc09d-5259-4d19-ab98-2cf90862bef2" volumeName="kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087541    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087552    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634" volumeName="kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087559    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3925cae-4e22-41e0-b510-7c5cec2bfd17" volumeName="kubernetes.io/empty-dir/f3925cae-4e22-41e0-b510-7c5cec2bfd17-audit-log" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087571    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3925cae-4e22-41e0-b510-7c5cec2bfd17" volumeName="kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-client-ca-bundle" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087578    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b9c7068-7163-4c61-8a7f-5ec789247fa7" volumeName="kubernetes.io/configmap/0b9c7068-7163-4c61-8a7f-5ec789247fa7-serviceca" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087595    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087603    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61cc09d-5259-4d19-ab98-2cf90862bef2" volumeName="kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087613    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" volumeName="kubernetes.io/projected/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-kube-api-access-mgsx5" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087624    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a979a593-d7fc-4659-a932-ca61a3801f27" volumeName="kubernetes.io/empty-dir/a979a593-d7fc-4659-a932-ca61a3801f27-tmp-dir" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087643    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebd094-601c-462b-a42c-89e9d6a27c2a" volumeName="kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087662    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46dd1c02-246a-41f2-bb6a-7523cc8b0436" volumeName="kubernetes.io/configmap/46dd1c02-246a-41f2-bb6a-7523cc8b0436-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087681    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634" volumeName="kubernetes.io/configmap/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087693    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31e88aa-14a5-4deb-a503-17b7f52cb4b9" volumeName="kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-env-overrides" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087704    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91513898-3858-4991-afad-cb5999ab98d0" volumeName="kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087715    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91513898-3858-4991-afad-cb5999ab98d0" volumeName="kubernetes.io/projected/91513898-3858-4991-afad-cb5999ab98d0-kube-api-access-hg9p5" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087726    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087736    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61cc09d-5259-4d19-ab98-2cf90862bef2" volumeName="kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087747    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634" volumeName="kubernetes.io/projected/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-kube-api-access-fswgb" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087759    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634" volumeName="kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087770    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31e88aa-14a5-4deb-a503-17b7f52cb4b9" volumeName="kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087781    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61317c8f-5a2e-4268-b085-df795ec4baa6" volumeName="kubernetes.io/projected/61317c8f-5a2e-4268-b085-df795ec4baa6-kube-api-access-l8km5" seLinuxMountContext=""
Oct 30 19:06:58.087756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087792    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61cc09d-5259-4d19-ab98-2cf90862bef2" volumeName="kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087803    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087815    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="558fbbeb-c828-4767-aecc-8c319ab91b35" volumeName="kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087828    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="558fbbeb-c828-4767-aecc-8c319ab91b35" volumeName="kubernetes.io/projected/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-api-access-5wdgm" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087839    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60953151-cd1e-4572-a9f1-a618fee9ef83" volumeName="kubernetes.io/secret/60953151-cd1e-4572-a9f1-a618fee9ef83-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087878    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087893    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3925cae-4e22-41e0-b510-7c5cec2bfd17" volumeName="kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087905    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91513898-3858-4991-afad-cb5999ab98d0" volumeName="kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087916    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087927    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9f6e91-3cab-4b47-9526-a50b62527a16" volumeName="kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087939    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbb1844b-cbea-4ef6-a608-340b7ae7fee5" volumeName="kubernetes.io/projected/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kube-api-access-tp66z" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087954    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebd094-601c-462b-a42c-89e9d6a27c2a" volumeName="kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087969    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.087995    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088029    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088040    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5232cfc3-47d5-46bb-8181-ecd05bc02348" volumeName="kubernetes.io/projected/5232cfc3-47d5-46bb-8181-ecd05bc02348-kube-api-access-mbk85" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088052    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088081    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d018801-0a52-45a6-a6c5-94692d11163a" volumeName="kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088092    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaa77658-47ce-47ee-b0a3-ec9dc492c3b3" volumeName="kubernetes.io/projected/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-kube-api-access-b2nnm" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088102    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ea8db5-8a84-4f64-a520-8089c8ee8231" volumeName="kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088114    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31e88aa-14a5-4deb-a503-17b7f52cb4b9" volumeName="kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-config" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088138    6777 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31e88aa-14a5-4deb-a503-17b7f52cb4b9" volumeName="kubernetes.io/secret/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088155    6777 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088177    6777 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088212    6777 factory.go:153] Registering CRI-O factory
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088228    6777 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088248    6777 factory.go:103] Registering Raw factory
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088263    6777 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:06:58.090775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.088598    6777 manager.go:319] Starting recovery of all containers
Oct 30 19:06:58.093142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.092915    6777 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:06:58.095980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.093369    6777 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:06:58.097189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.097168    6777 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:06:58.099576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.098823    6777 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:06:58.102912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.102872    6777 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:06:58.104356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.104337    6777 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:06:58.104356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.104356    6777 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:06:58.104495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.104377    6777 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:06:58.104495 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:06:58.104418    6777 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:06:58.108402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.108383    6777 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:06:58.120090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.120029    6777 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="1085d7272173633864c6c822849ecc3645a82ebc6d786f407fda22709e961eec" exitCode=0
Oct 30 19:06:58.120090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.120088    6777 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="cfc6344599fb013241c4c7c55bb8d37bf21f5fc9686277554c4156ebe6a8ba88" exitCode=0
Oct 30 19:06:58.120241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.120100    6777 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="1ac55b136d00118d482a1e118bea33965c64ba627a2d958cba56b644e6ed122d" exitCode=0
Oct 30 19:06:58.120241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.120109    6777 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="7672a8c42cf626344a4deaec11d8becbd028f5a38b5cb9ab65b63bdc33182327" exitCode=0
Oct 30 19:06:58.120241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.120118    6777 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="16e6103eabf6954a78bba6ca09877442b8eaddd5828fe85d949f243a929490be" exitCode=0
Oct 30 19:06:58.120241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.120126    6777 generic.go:356] "Generic (PLEG): container finished" podID="c6585f59-5f16-4766-a3a2-e8aeac2c4bcb" containerID="9e024d2561088966412594a34df6b9eb77f51f3233e545f2172ca1cb473336e7" exitCode=0
Oct 30 19:06:58.129979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.129955    6777 generic.go:356] "Generic (PLEG): container finished" podID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerID="477c9e680d4f90899fc2330471567ed48624dd5e6a60b5216daab4e08a9a50f4" exitCode=0
Oct 30 19:06:58.132218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.132196    6777 generic.go:356] "Generic (PLEG): container finished" podID="91513898-3858-4991-afad-cb5999ab98d0" containerID="960242d5933e70aa8b921be6625d707285a1beebe2c3b36cf99c1aee2b7a5a9a" exitCode=0
Oct 30 19:06:58.142145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.142124    6777 generic.go:356] "Generic (PLEG): container finished" podID="9d018801-0a52-45a6-a6c5-94692d11163a" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02" exitCode=0
Oct 30 19:06:58.170345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.170318    6777 generic.go:356] "Generic (PLEG): container finished" podID="617112a22c78eeb8b0062c43970103d2" containerID="64405d5dff437f04cf8c54a0467494cc4cd3437fff3df0018c9ec3af1d6b97dd" exitCode=0
Oct 30 19:06:58.204539 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:06:58.204476    6777 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 19:06:58.214563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.214544    6777 manager.go:324] Recovery completed
Oct 30 19:06:58.241454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241419    6777 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:06:58.241454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241447    6777 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:06:58.241626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241469    6777 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:06:58.241626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241592    6777 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:06:58.241626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241603    6777 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:06:58.241626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241619    6777 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:06:58.241626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241625    6777 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:06:58.241795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241631    6777 policy_none.go:50] "Start"
Oct 30 19:06:58.241795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241641    6777 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:06:58.241795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241650    6777 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:06:58.241795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241772    6777 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:06:58.241795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241779    6777 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:06:58.241795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.241784    6777 policy_none.go:44] "Start"
Oct 30 19:06:58.246751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.246722    6777 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:06:58.249418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.249404    6777 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:06:58.249944 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:06:58.249431    6777 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:06:58.250009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.249946    6777 server.go:87] "Starting device plugin registration server"
Oct 30 19:06:58.250236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.250225    6777 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:06:58.250285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.250236    6777 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:06:58.250330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.250305    6777 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:06:58.250437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.250391    6777 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:06:58.250437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.250431    6777 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:06:58.250437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.250438    6777 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:06:58.250640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.250487    6777 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:06:58.250421404Z","UUID":"d92c2a2b-0d0b-4329-ba77-af031205550b","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:06:58.252159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.252138    6777 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:06:58.252288 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:06:58.252157    6777 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:06:58.252464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.252179    6777 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:06:58.350773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.350731    6777 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:06:58.352685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.352665    6777 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:06:58.352781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.352697    6777 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:06:58.352781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.352712    6777 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:06:58.352781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.352760    6777 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.366991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.366952    6777 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.367080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.367035    6777 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.405014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.404969    6777 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"]
Oct 30 19:06:58.405616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.405594    6777 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.405730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.405669    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"617112a22c78eeb8b0062c43970103d2","Type":"ContainerStarted","Data":"f07cecb5226ed8c7e0b7aed9c278386f2cf37f1475be14f7ef1965c679c0f5b2"}
Oct 30 19:06:58.405730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.405708    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"617112a22c78eeb8b0062c43970103d2","Type":"ContainerDied","Data":"64405d5dff437f04cf8c54a0467494cc4cd3437fff3df0018c9ec3af1d6b97dd"}
Oct 30 19:06:58.405730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.405722    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"617112a22c78eeb8b0062c43970103d2","Type":"ContainerStarted","Data":"8389b33f6f123f5ffab77d0a7ac3f81ba9b7460ebc6892248ab3bdb028ae42eb"}
Oct 30 19:06:58.405823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.405734    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"b20bfab463977c229c9db20c57c04c0b","Type":"ContainerStarted","Data":"cba1b9dd034e4082bb1912ee3b0ceb0033c2559f23eea6cf75113ff342d3db38"}
Oct 30 19:06:58.405823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.405747    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal" event={"ID":"b20bfab463977c229c9db20c57c04c0b","Type":"ContainerStarted","Data":"4764481df7a51d46ba985f6d991c7cb4ba0bc7a616154ce8f03af416e9b3d0ba"}
Oct 30 19:06:58.405879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.405852    6777 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.420383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.420364    6777 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:06:58.420457 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:06:58.420406    6777 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.421389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.421378    6777 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:06:58.421427 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:06:58.421402    6777 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.498805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.498732    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b20bfab463977c229c9db20c57c04c0b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"b20bfab463977c229c9db20c57c04c0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.498805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.498777    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.498805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.498800    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.599312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.599273    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b20bfab463977c229c9db20c57c04c0b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"b20bfab463977c229c9db20c57c04c0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.599312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.599316    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.599485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.599341    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.599485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.599387    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.599485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.599398    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/617112a22c78eeb8b0062c43970103d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"617112a22c78eeb8b0062c43970103d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:58.599485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:58.599387    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b20bfab463977c229c9db20c57c04c0b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal\" (UID: \"b20bfab463977c229c9db20c57c04c0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:59.068173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.068137    6777 apiserver.go:52] "Watching apiserver"
Oct 30 19:06:59.082029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.082004    6777 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:06:59.082905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.082874    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0","openshift-ovn-kubernetes/ovnkube-node-ldfm8","openshift-console/console-f645b69cc-d5dm6","openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb","openshift-monitoring/alertmanager-main-0","openshift-insights/insights-runtime-extractor-vtpvd","kube-system/konnectivity-agent-24ljs","openshift-cluster-node-tuning-operator/tuned-ml5wz","openshift-image-registry/node-ca-9b5ds","openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn","openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs","openshift-monitoring/metrics-server-665cc87487-rzd5l","openshift-network-diagnostics/network-check-target-k4plb","kube-system/global-pull-secret-syncer-tvwkt","openshift-dns/node-resolver-829qt","openshift-monitoring/node-exporter-tsr7c","openshift-console/console-6474bf5b46-vc9zh","kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal","openshift-monitoring/thanos-querier-77856dc4d7-j9gps","openshift-multus/multus-additional-cni-plugins-5bdll","openshift-multus/network-metrics-daemon-tbsh8","openshift-network-operator/iptables-alerter-ctfbf","openshift-multus/multus-v9ps7","openshift-dns/dns-default-vpj4h","openshift-ingress-canary/ingress-canary-m89p4"]
Oct 30 19:06:59.089158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.089090    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.089297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.089198    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.089297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.089277    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.089380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.089315    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.089409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.089390    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:06:59.089436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.089410    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.090089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.089966    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.090089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.089970    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.090089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.090020    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.090089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.090007    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.090089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.090022    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.090362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.090279    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.090410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.090372    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:06:59.090456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.090425    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.092066 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092036    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:06:59.092307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092286    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:06:59.092307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092290    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ngr9b\""
Oct 30 19:06:59.092459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092290    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:06:59.092459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092399    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:06:59.092459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092410    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gkzrx\""
Oct 30 19:06:59.092602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092548    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:06:59.092602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092582    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w2h58\""
Oct 30 19:06:59.092602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092582    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xd9sf\""
Oct 30 19:06:59.092602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092602    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fljvz\""
Oct 30 19:06:59.092768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092624    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jks97\""
Oct 30 19:06:59.092768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.092630    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:06:59.093269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.093250    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-m74bb\""
Oct 30 19:06:59.093370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.093283    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:06:59.093370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.093333    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hsp67\""
Oct 30 19:06:59.093710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.093694    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.093879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.093864    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.093993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.093980    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xlxzb\""
Oct 30 19:06:59.094068 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.093992    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:06:59.095205 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095188    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.095324 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095195    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:06:59.095460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095447    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4z6gp\""
Oct 30 19:06:59.095512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095496    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.095546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095523    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:06:59.095595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095584    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:06:59.095663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095652    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-89jvk\""
Oct 30 19:06:59.095715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095704    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:06:59.095748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095739    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:06:59.095845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095831    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tc7v2\""
Oct 30 19:06:59.095952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.095940    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.096132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.096117    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.096379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.096364    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:06:59.096534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.096520    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:06:59.096744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.096730    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:06:59.096799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.096789    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fsc2n\""
Oct 30 19:06:59.096846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.096837    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:06:59.096940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.096930    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6h85f\""
Oct 30 19:06:59.100833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.100807    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b9c7068-7163-4c61-8a7f-5ec789247fa7-host\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:06:59.100932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.100846    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-lib-modules\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.100932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.100898    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b9c7068-7163-4c61-8a7f-5ec789247fa7-serviceca\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:06:59.100932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.100925    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6wg\" (UniqueName: \"kubernetes.io/projected/0b9c7068-7163-4c61-8a7f-5ec789247fa7-kube-api-access-dw6wg\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:06:59.101080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.100954    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-kubernetes\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.100979    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysconfig\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101003    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101045    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-systemd\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101095    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-run\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101121    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-sys\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101156    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-host\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101256    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-tuned\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101300    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-modprobe-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101330    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-conf\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101343    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-tuned\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101350    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-var-lib-kubelet\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101453    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-tmp\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101477    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbk85\" (UniqueName: \"kubernetes.io/projected/5232cfc3-47d5-46bb-8181-ecd05bc02348-kube-api-access-mbk85\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.101595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101578    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b9c7068-7163-4c61-8a7f-5ec789247fa7-serviceca\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:06:59.101768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.101591    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5232cfc3-47d5-46bb-8181-ecd05bc02348-tmp\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.111983 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.111955    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6wg\" (UniqueName: \"kubernetes.io/projected/0b9c7068-7163-4c61-8a7f-5ec789247fa7-kube-api-access-dw6wg\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:06:59.112423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.112404    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbk85\" (UniqueName: \"kubernetes.io/projected/5232cfc3-47d5-46bb-8181-ecd05bc02348-kube-api-access-mbk85\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.178785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.178752    6777 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:59.178958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.178852    6777 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:59.185918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.185887    6777 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:06:59.191353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.191334    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.191438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.191380    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.191490 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.191449    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.191940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.191927    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.192184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.192156    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 19:06:59.192247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.192234    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 19:06:59.192601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.192589    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 19:06:59.192649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.192636    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:06:59.192861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.192848    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 19:06:59.192926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.192915    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 19:06:59.193037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.193025    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:06:59.193436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.193424    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:06:59.193601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.193588    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:06:59.193642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.193625    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:06:59.193717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.193697    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:06:59.197888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.197872    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 19:06:59.197975 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.197895    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:06:59.197975 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.197911    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:06:59.202074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202033    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-sys\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.202162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202082    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.202162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202128    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9p5\" (UniqueName: \"kubernetes.io/projected/91513898-3858-4991-afad-cb5999ab98d0-kube-api-access-hg9p5\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.202280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202184    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.202280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202213    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.202280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202237    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-modprobe-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.202280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202262    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-conf\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.202484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202306    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp66z\" (UniqueName: \"kubernetes.io/projected/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kube-api-access-tp66z\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.202484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202346    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-kubelet\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.202484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202364    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.202484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202372    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.202484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202400    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.202484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202424    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.202484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202421    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-modprobe-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.202484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202424    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-conf\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.202484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202476    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202515    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-systemd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202544    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-var-lib-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202587    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-env-overrides\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202629    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ac8869-277b-44e4-a31a-bde13751efd9-tmp-dir\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202652    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202712    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202730    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ac8869-277b-44e4-a31a-bde13751efd9-tmp-dir\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202751    6777 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202783    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-wtmp\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202796    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202818    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202843    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202864    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52v6\" (UniqueName: \"kubernetes.io/projected/38ac8869-277b-44e4-a31a-bde13751efd9-kube-api-access-g52v6\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:06:59.202908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202886    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202915    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-env-overrides\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.202987    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203045    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203085    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203106    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqhq\" (UniqueName: \"kubernetes.io/projected/46dd1c02-246a-41f2-bb6a-7523cc8b0436-kube-api-access-csqhq\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203168    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysconfig\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:06:59.203182    6777 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203201    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203226    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-cni-binary-copy\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203248    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tmhg\" (UniqueName: \"kubernetes.io/projected/bf9f6e91-3cab-4b47-9526-a50b62527a16-kube-api-access-9tmhg\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203290    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203311    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l5r\" (UniqueName: \"kubernetes.io/projected/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-kube-api-access-z7l5r\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203338    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203364    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysctl-d\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203363    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-socket-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203427    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-cni-binary-copy\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203514    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203539    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvjr\" (UniqueName: \"kubernetes.io/projected/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-api-access-qhvjr\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203570    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203578    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203597    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203643    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/baec05dd-6d69-41d9-ad73-4179b17ba8f4-original-pull-secret\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:59.203665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203670    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b9c7068-7163-4c61-8a7f-5ec789247fa7-host\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203700    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46dd1c02-246a-41f2-bb6a-7523cc8b0436-iptables-alerter-script\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203735    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-cnibin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203785    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203829    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203847    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203875    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203886    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203920    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203949    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjd94\" (UniqueName: \"kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203958    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.203977    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204006    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-sysconfig\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204003    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60953151-cd1e-4572-a9f1-a618fee9ef83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204085    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-script-lib\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204098    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b9c7068-7163-4c61-8a7f-5ec789247fa7-host\") pod \"node-ca-9b5ds\" (UID: \"0b9c7068-7163-4c61-8a7f-5ec789247fa7\") " pod="openshift-image-registry/node-ca-9b5ds"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204116    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dd1c02-246a-41f2-bb6a-7523cc8b0436-host-slash\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204283    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204336    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-k8s-cni-cncf-io\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204371    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204396    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-multus\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204411    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60953151-cd1e-4572-a9f1-a618fee9ef83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204422    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204417    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-script-lib\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204447    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-systemd-units\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204479    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61317c8f-5a2e-4268-b085-df795ec4baa6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204489    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204522    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204559    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46dd1c02-246a-41f2-bb6a-7523cc8b0436-iptables-alerter-script\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204603    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204650    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204659    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a979a593-d7fc-4659-a932-ca61a3801f27-config-volume\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204667    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61317c8f-5a2e-4268-b085-df795ec4baa6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204703    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-cert\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204763    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-dbus\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204766    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204762    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204804    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsx5\" (UniqueName: \"kubernetes.io/projected/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-kube-api-access-mgsx5\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204829    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38ac8869-277b-44e4-a31a-bde13751efd9-hosts-file\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204847    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a979a593-d7fc-4659-a932-ca61a3801f27-config-volume\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204857    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204890    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrmg\" (UniqueName: \"kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204906    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204923    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.204912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.204940    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-node-log\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205211    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-sys-fs\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205387    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205448    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205488    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205550    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205573    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9ctdn\""
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205590    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-lib-modules\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205718    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205742    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205785    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205892    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a979a593-d7fc-4659-a932-ca61a3801f27-tmp-dir\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205929    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cnibin\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.205963    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-kubelet-config\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.206119    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-lib-modules\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.206356    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.206436    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.206567    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.206615    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.206842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.206711    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207030    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207101    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnw4\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207140    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207147    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207213    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mzlzt\""
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207262    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207462    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207473    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207615    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a979a593-d7fc-4659-a932-ca61a3801f27-tmp-dir\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207674    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.207766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207740    6777 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207770    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:06:59.207791    6777 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-19.us-east-2.compute.internal"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207833    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207908    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207955    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdgm\" (UniqueName: \"kubernetes.io/projected/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-api-access-5wdgm\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208012    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-metrics-server-audit-profiles\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208046    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-daemon-config\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208167    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208184    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-multus-certs\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208219    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-root\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208249    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-slash\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208271    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sg4ml\""
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208278    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207795    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f6l0lstb29pus\""
Oct 30 19:06:59.208313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208311    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-metrics-client-ca\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208341    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208374    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208405    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmwz8\" (UniqueName: \"kubernetes.io/projected/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-kube-api-access-bmwz8\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208431    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208432    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60953151-cd1e-4572-a9f1-a618fee9ef83-crio-socket\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208462    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fd7q8\""
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208467    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208559    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208584    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208590    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswgb\" (UniqueName: \"kubernetes.io/projected/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-kube-api-access-fswgb\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208619    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-system-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208653    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208687    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208719    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208751    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208771    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208823    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208888    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-textfile\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208909    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:06:59.209046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208916    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:06:59.209914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.209269    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n985b\""
Oct 30 19:06:59.209914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.209290    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:06:59.209914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.207980    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wdms4\""
Oct 30 19:06:59.209914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.209532    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xxr6f\""
Oct 30 19:06:59.209914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.209593    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:06:59.209914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.209745    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-metrics-server-audit-profiles\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.209914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.209829    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 19:06:59.210225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.209933    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bbsbt\""
Oct 30 19:06:59.210225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.209957    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-daemon-config\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.210225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210041    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:06:59.210350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210283    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.210496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210474    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:06:59.210563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.208775    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-textfile\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.210563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210546    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.210741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210562    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91513898-3858-4991-afad-cb5999ab98d0-metrics-client-ca\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.210741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210583    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-kubelet\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.210741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210629    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 19:06:59.210741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210634    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-metrics-client-ca\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.210741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210674    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-systemd\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.210741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210707    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60953151-cd1e-4572-a9f1-a618fee9ef83-data-volume\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.210741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210725    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:06:59.210741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210739    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 19:06:59.210741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210740    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-grpc-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210774    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-bin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210804    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-conf-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210817    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210829    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-etc-kubernetes\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210858    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210888    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-system-cni-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210918    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-llnck\""
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210994    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210553    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211171    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7bpwj\""
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211307    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211353    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 19:06:59.211403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.210918    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-log-socket\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211434    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-bin\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211484    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211383    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211771    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ams4o7htr81ts\""
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211794    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-metrics-client-ca\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211811    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-systemd\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211833    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60953151-cd1e-4572-a9f1-a618fee9ef83-data-volume\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211841    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211791    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211928    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 19:06:59.212035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211948    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.212482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.212135    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-grpc-tls\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.212482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.212356    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.212847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.212735    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.212885    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.212893    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.212914    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.212914    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.212932    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.211901    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-client-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213011    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-etc-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213040    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213088    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213112    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213126    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213172    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgsn\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213202    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a979a593-d7fc-4659-a932-ca61a3801f27-metrics-tls\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213227    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/558fbbeb-c828-4767-aecc-8c319ab91b35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213257    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-netd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213264    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213282    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-konnectivity-ca\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213292    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213316    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213344    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213306    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-hostroot\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213382    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5b3rasg0scipm\""
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213397    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-client-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213426    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213427    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/558fbbeb-c828-4767-aecc-8c319ab91b35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213479    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213497    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8km5\" (UniqueName: \"kubernetes.io/projected/61317c8f-5a2e-4268-b085-df795ec4baa6-kube-api-access-l8km5\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213496    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213511    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-konnectivity-ca\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213521    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-var-lib-kubelet\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.213561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213548    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-device-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213738    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213774    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213795    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213805    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-tls\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213828    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a979a593-d7fc-4659-a932-ca61a3801f27-metrics-tls\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213868    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213904    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqhq\" (UniqueName: \"kubernetes.io/projected/46dd1c02-246a-41f2-bb6a-7523cc8b0436-kube-api-access-csqhq\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213965    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-var-lib-kubelet\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213835    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r27l\" (UniqueName: \"kubernetes.io/projected/77ff1252-ac70-4fad-a824-99f3c1eb860e-kube-api-access-6r27l\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213982    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9p5\" (UniqueName: \"kubernetes.io/projected/91513898-3858-4991-afad-cb5999ab98d0-kube-api-access-hg9p5\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.213991    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214026    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52v6\" (UniqueName: \"kubernetes.io/projected/38ac8869-277b-44e4-a31a-bde13751efd9-kube-api-access-g52v6\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214083    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214095    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2nnm\" (UniqueName: \"kubernetes.io/projected/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-kube-api-access-b2nnm\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214150    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-tls\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214194    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214214    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214242    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214249    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214262    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214292    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-kubernetes\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214321    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214350    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214374    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61317c8f-5a2e-4268-b085-df795ec4baa6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214380    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214377    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214424    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-etc-kubernetes\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214456    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrl6\" (UniqueName: \"kubernetes.io/projected/a979a593-d7fc-4659-a932-ca61a3801f27-kube-api-access-frrl6\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214466    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214484    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214510    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214533    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214556    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214561    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-registration-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214565    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214588    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-sys\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214614    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214665    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tmhg\" (UniqueName: \"kubernetes.io/projected/bf9f6e91-3cab-4b47-9526-a50b62527a16-kube-api-access-9tmhg\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214783    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-host\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214834    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp66z\" (UniqueName: \"kubernetes.io/projected/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kube-api-access-tp66z\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214815    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-socket-dir-parent\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214872    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-sys\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214877    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjd94\" (UniqueName: \"kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214882    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-tls\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214838    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-metrics-certs\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214933    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7l5r\" (UniqueName: \"kubernetes.io/projected/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-kube-api-access-z7l5r\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.214995    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215000    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215048    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-netns\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215090    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-ovn\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215111    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvjr\" (UniqueName: \"kubernetes.io/projected/60953151-cd1e-4572-a9f1-a618fee9ef83-kube-api-access-qhvjr\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215090    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215143    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-cert\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215137    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-agent-certs\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215144    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215267    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215277    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-host\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215313    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215327    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-tls\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215353    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215384    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b26e19e-b8e9-496a-bfcb-6cd5add61e6e-agent-certs\") pod \"konnectivity-agent-24ljs\" (UID: \"1b26e19e-b8e9-496a-bfcb-6cd5add61e6e\") " pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215417    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovn-node-metrics-cert\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215502    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5px2\" (UniqueName: \"kubernetes.io/projected/6e38a783-1270-4703-a97c-cabdbe1c50ce-kube-api-access-b5px2\") pod \"network-check-target-k4plb\" (UID: \"6e38a783-1270-4703-a97c-cabdbe1c50ce\") " pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215505    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215539    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215544    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215565    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-netns\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215695    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215884    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215933    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7vwl\" (UniqueName: \"kubernetes.io/projected/f3925cae-4e22-41e0-b510-7c5cec2bfd17-kube-api-access-l7vwl\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.215977    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77ff1252-ac70-4fad-a824-99f3c1eb860e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216006    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-tmp\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216138    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216210    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3925cae-4e22-41e0-b510-7c5cec2bfd17-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216213    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovn-node-metrics-cert\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216279    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77ff1252-ac70-4fad-a824-99f3c1eb860e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216325    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216371    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216326    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-tmp\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216397    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216421    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrmg\" (UniqueName: \"kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216432    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216459    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216499    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216486    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-client-certs\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216540    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-config\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216569    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f3925cae-4e22-41e0-b510-7c5cec2bfd17-audit-log\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216577    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert\") pod \"console-6474bf5b46-vc9zh\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") " pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216586    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216591    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-run\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.216579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216645    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp\") pod \"console-f645b69cc-d5dm6\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") " pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:06:59.219147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216678    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.219147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216696    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f3925cae-4e22-41e0-b510-7c5cec2bfd17-audit-log\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.219147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216685    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5232cfc3-47d5-46bb-8181-ecd05bc02348-run\") pod \"tuned-ml5wz\" (UID: \"5232cfc3-47d5-46bb-8181-ecd05bc02348\") " pod="openshift-cluster-node-tuning-operator/tuned-ml5wz"
Oct 30 19:06:59.219147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216715    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f3925cae-4e22-41e0-b510-7c5cec2bfd17-secret-metrics-server-client-certs\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.219147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.216747    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-ovnkube-config\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.219147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.217927    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.219147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.218286    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsx5\" (UniqueName: \"kubernetes.io/projected/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-kube-api-access-mgsx5\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.219147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.218943    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.220717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.220701    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnw4\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.221492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.221475    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdgm\" (UniqueName: \"kubernetes.io/projected/558fbbeb-c828-4767-aecc-8c319ab91b35-kube-api-access-5wdgm\") pod \"kube-state-metrics-5dc8d7d89b-976bs\" (UID: \"558fbbeb-c828-4767-aecc-8c319ab91b35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-976bs"
Oct 30 19:06:59.221619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.221607    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:59.222980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.222944    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmwz8\" (UniqueName: \"kubernetes.io/projected/ceeebf2b-9b37-4988-b6c1-b8b41e09449a-kube-api-access-bmwz8\") pod \"network-metrics-daemon-tbsh8\" (UID: \"ceeebf2b-9b37-4988-b6c1-b8b41e09449a\") " pod="openshift-multus/network-metrics-daemon-tbsh8"
Oct 30 19:06:59.223331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.223315    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswgb\" (UniqueName: \"kubernetes.io/projected/d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634-kube-api-access-fswgb\") pod \"thanos-querier-77856dc4d7-j9gps\" (UID: \"d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634\") " pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:06:59.225110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.225091    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrl6\" (UniqueName: \"kubernetes.io/projected/a979a593-d7fc-4659-a932-ca61a3801f27-kube-api-access-frrl6\") pod \"dns-default-vpj4h\" (UID: \"a979a593-d7fc-4659-a932-ca61a3801f27\") " pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:06:59.225347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.225328    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8km5\" (UniqueName: \"kubernetes.io/projected/61317c8f-5a2e-4268-b085-df795ec4baa6-kube-api-access-l8km5\") pod \"openshift-state-metrics-7bd7489957-6lpsn\" (UID: \"61317c8f-5a2e-4268-b085-df795ec4baa6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lpsn"
Oct 30 19:06:59.225450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.225424    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2nnm\" (UniqueName: \"kubernetes.io/projected/aaa77658-47ce-47ee-b0a3-ec9dc492c3b3-kube-api-access-b2nnm\") pod \"ingress-canary-m89p4\" (UID: \"aaa77658-47ce-47ee-b0a3-ec9dc492c3b3\") " pod="openshift-ingress-canary/ingress-canary-m89p4"
Oct 30 19:06:59.225549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.225529    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r27l\" (UniqueName: \"kubernetes.io/projected/77ff1252-ac70-4fad-a824-99f3c1eb860e-kube-api-access-6r27l\") pod \"monitoring-plugin-77f4f76594-6mh9p\" (UID: \"77ff1252-ac70-4fad-a824-99f3c1eb860e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:06:59.225587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.225541    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgsn\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn\") pod \"alertmanager-main-0\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:06:59.225904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.225889    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7vwl\" (UniqueName: \"kubernetes.io/projected/f3925cae-4e22-41e0-b510-7c5cec2bfd17-kube-api-access-l7vwl\") pod \"metrics-server-665cc87487-rzd5l\" (UID: \"f3925cae-4e22-41e0-b510-7c5cec2bfd17\") " pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:06:59.317563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317531    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dd1c02-246a-41f2-bb6a-7523cc8b0436-host-slash\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:06:59.317563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317564    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-k8s-cni-cncf-io\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317580    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-multus\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317596    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-systemd-units\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317635    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-multus\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317644    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-k8s-cni-cncf-io\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317644    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dd1c02-246a-41f2-bb6a-7523cc8b0436-host-slash\") pod \"iptables-alerter-ctfbf\" (UID: \"46dd1c02-246a-41f2-bb6a-7523cc8b0436\") " pod="openshift-network-operator/iptables-alerter-ctfbf"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317689    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-dbus\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317708    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38ac8869-277b-44e4-a31a-bde13751efd9-hosts-file\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317726    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-node-log\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317732    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-systemd-units\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317740    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-sys-fs\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317779    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38ac8869-277b-44e4-a31a-bde13751efd9-hosts-file\") pod \"node-resolver-829qt\" (UID: \"38ac8869-277b-44e4-a31a-bde13751efd9\") " pod="openshift-dns/node-resolver-829qt"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317788    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-node-log\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317805    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-dbus\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317813    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317837    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317795    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-sys-fs\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317850    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317854    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cnibin\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317893    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317914    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-kubelet-config\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317893    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-cnibin\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317942    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-multus-certs\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317956    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-root\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317987    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/baec05dd-6d69-41d9-ad73-4179b17ba8f4-kubelet-config\") pod \"global-pull-secret-syncer-tvwkt\" (UID: \"baec05dd-6d69-41d9-ad73-4179b17ba8f4\") " pod="kube-system/global-pull-secret-syncer-tvwkt"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.317979    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-slash\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318014    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-root\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318026    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-multus-certs\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318039    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-slash\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318026    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60953151-cd1e-4572-a9f1-a618fee9ef83-crio-socket\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318083    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60953151-cd1e-4572-a9f1-a618fee9ef83-crio-socket\") pod \"insights-runtime-extractor-vtpvd\" (UID: \"60953151-cd1e-4572-a9f1-a618fee9ef83\") " pod="openshift-insights/insights-runtime-extractor-vtpvd"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318092    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318115    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-system-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.318123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318134    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-kubelet\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318150    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-bin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318164    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-conf-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318178    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-etc-kubernetes\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318193    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-system-cni-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318202    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-kubelet\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318194    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-system-cni-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318224    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-cni-bin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318234    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-conf-dir\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318230    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-system-cni-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318232    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-log-socket\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318235    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318260    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-log-socket\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318259    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-etc-kubernetes\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318273    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-bin\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318308    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-bin\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318313    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-etc-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318332    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-netd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318347    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-hostroot\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318362    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-device-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318362    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-etc-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318382    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318394    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-cni-netd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318404    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-hostroot\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318418    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-device-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318407    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-registration-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318453    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-socket-dir-parent\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318431    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318456    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-registration-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318468    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-netns\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318486    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-host-run-netns\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318500    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-ovn\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318519    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-multus-socket-dir-parent\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318524    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318547    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318555    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-ovn\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318566    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-netns\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318578    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318589    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-sys\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318593    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318602    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-run-netns\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318618    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-sys\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318627    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-kubelet\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318655    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-systemd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318677    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-host-var-lib-kubelet\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318679    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-var-lib-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318698    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-run-systemd\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318712    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31e88aa-14a5-4deb-a503-17b7f52cb4b9-var-lib-openvswitch\") pod \"ovnkube-node-ldfm8\" (UID: \"f31e88aa-14a5-4deb-a503-17b7f52cb4b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318717    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-wtmp\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318741    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-socket-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318758    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-cnibin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318774    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318798    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbb1844b-cbea-4ef6-a608-340b7ae7fee5-socket-dir\") pod \"aws-ebs-csi-driver-node-rf8cb\" (UID: \"cbb1844b-cbea-4ef6-a608-340b7ae7fee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rf8cb"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318821    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9f6e91-3cab-4b47-9526-a50b62527a16-cnibin\") pod \"multus-v9ps7\" (UID: \"bf9f6e91-3cab-4b47-9526-a50b62527a16\") " pod="openshift-multus/multus-v9ps7"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318833    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91513898-3858-4991-afad-cb5999ab98d0-node-exporter-wtmp\") pod \"node-exporter-tsr7c\" (UID: \"91513898-3858-4991-afad-cb5999ab98d0\") " pod="openshift-monitoring/node-exporter-tsr7c"
Oct 30 19:06:59.319040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:06:59.318848    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6585f59-5f16-4766-a3a2-e8aeac2c4bcb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bdll\" (UID: \"c6585f59-5f16-4766-a3a2-e8aeac2c4bcb\") " pod="openshift-multus/multus-additional-cni-plugins-5bdll"
Oct 30 19:07:00.065101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:00.065054    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-665cc87487-rzd5l"
Oct 30 19:07:00.132192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:00.132148    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vpj4h"
Oct 30 19:07:00.992288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:00.992248    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6mh9p"
Oct 30 19:07:01.563158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:01.563124    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77856dc4d7-j9gps"
Oct 30 19:07:01.749743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:01.749716    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:07:03.534492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:03.534463    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:05.227979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:05.227945    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:07:06.653884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:06.653858    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:07:06.654256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:06.654086    6777 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:07:08.256587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:08.256548    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:07:11.711029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:11.710928    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-24ljs"
Oct 30 19:07:12.471156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:12.471123    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:07:12.471476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:12.471424    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="alertmanager" containerID="cri-o://30cff2d19c3b93498e05f0af04a666158378dc198da2853aaae5b934feabb715" gracePeriod=120
Oct 30 19:07:12.471618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:12.471499    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy-metric" containerID="cri-o://91696330db95516eba6dc9dd915b57b425c7232ed18bf3910a9597d4e7f42770" gracePeriod=120
Oct 30 19:07:12.471618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:12.471571    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="config-reloader" containerID="cri-o://818d8e2701017f58ea2fd0a9cf8b328205d5db465ccd3af91d5d8fc0afc7f017" gracePeriod=120
Oct 30 19:07:12.471720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:12.471602    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy" containerID="cri-o://d3495f64628493e7d3741f8321cca49762e390ab26564d8df44c3d089a349490" gracePeriod=120
Oct 30 19:07:12.471720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:12.471693    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy-web" containerID="cri-o://63257b9f7a0346d3bf02b9d3cf07e83134f06fb36f0a62233ca557017218890d" gracePeriod=120
Oct 30 19:07:12.471967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:12.471929    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="prom-label-proxy" containerID="cri-o://17dc037d9f19a96e1c66f1097ad74aa4062b5403bb9860cc39b1ae39ac82ac82" gracePeriod=120
Oct 30 19:07:13.233840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.233806    6777 generic.go:356] "Generic (PLEG): container finished" podID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerID="17dc037d9f19a96e1c66f1097ad74aa4062b5403bb9860cc39b1ae39ac82ac82" exitCode=0
Oct 30 19:07:13.233840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.233834    6777 generic.go:356] "Generic (PLEG): container finished" podID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerID="91696330db95516eba6dc9dd915b57b425c7232ed18bf3910a9597d4e7f42770" exitCode=0
Oct 30 19:07:13.233840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.233841    6777 generic.go:356] "Generic (PLEG): container finished" podID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerID="d3495f64628493e7d3741f8321cca49762e390ab26564d8df44c3d089a349490" exitCode=0
Oct 30 19:07:13.233840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.233847    6777 generic.go:356] "Generic (PLEG): container finished" podID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerID="818d8e2701017f58ea2fd0a9cf8b328205d5db465ccd3af91d5d8fc0afc7f017" exitCode=0
Oct 30 19:07:13.233840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.233852    6777 generic.go:356] "Generic (PLEG): container finished" podID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerID="30cff2d19c3b93498e05f0af04a666158378dc198da2853aaae5b934feabb715" exitCode=0
Oct 30 19:07:13.234350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.233886    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerDied","Data":"17dc037d9f19a96e1c66f1097ad74aa4062b5403bb9860cc39b1ae39ac82ac82"}
Oct 30 19:07:13.234350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.233982    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerDied","Data":"91696330db95516eba6dc9dd915b57b425c7232ed18bf3910a9597d4e7f42770"}
Oct 30 19:07:13.234350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.234008    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerDied","Data":"d3495f64628493e7d3741f8321cca49762e390ab26564d8df44c3d089a349490"}
Oct 30 19:07:13.234350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.234034    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerDied","Data":"818d8e2701017f58ea2fd0a9cf8b328205d5db465ccd3af91d5d8fc0afc7f017"}
Oct 30 19:07:13.234350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:13.234075    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerDied","Data":"30cff2d19c3b93498e05f0af04a666158378dc198da2853aaae5b934feabb715"}
Oct 30 19:07:13.637178 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:13.637147    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:07:14.242760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.242723    6777 generic.go:356] "Generic (PLEG): container finished" podID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerID="63257b9f7a0346d3bf02b9d3cf07e83134f06fb36f0a62233ca557017218890d" exitCode=0
Oct 30 19:07:14.243132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.242781    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerDied","Data":"63257b9f7a0346d3bf02b9d3cf07e83134f06fb36f0a62233ca557017218890d"}
Oct 30 19:07:14.243132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.242855    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09ea8db5-8a84-4f64-a520-8089c8ee8231","Type":"ContainerDied","Data":"91ec2d72013cd4defd209ccca364d3c6550f52a3e42234c7a8772b296a560ee0"}
Oct 30 19:07:14.243132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.242896    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91ec2d72013cd4defd209ccca364d3c6550f52a3e42234c7a8772b296a560ee0"
Oct 30 19:07:14.256385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.256360    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:14.360436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360344    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360416    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360487    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360526    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360570    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360608    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360672    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360710    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360751    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360753    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:14.360854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360789    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.360854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360821    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.361244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360862    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.361244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.360905    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web\") pod \"09ea8db5-8a84-4f64-a520-8089c8ee8231\" (UID: \"09ea8db5-8a84-4f64-a520-8089c8ee8231\") "
Oct 30 19:07:14.361244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.361166    6777 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.361244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.361215    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:14.361607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.361579    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:14.364270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.364207    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:14.364564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.364524    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "kube-api-access-2xgsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:14.365232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.365206    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:14.365322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.365283    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:14.365539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.365506    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:14.365539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.365528    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:14.365670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.365616    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:14.366002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.365977    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:14.369017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.368828    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:14.389621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.389588    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config" pod "09ea8db5-8a84-4f64-a520-8089c8ee8231" (UID: "09ea8db5-8a84-4f64-a520-8089c8ee8231"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:14.462010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.461973    6777 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-web-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462007    6777 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-alertmanager-main-db\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462018    6777 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-tls-assets\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462028    6777 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462037    6777 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-out\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462047    6777 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09ea8db5-8a84-4f64-a520-8089c8ee8231-metrics-client-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462077    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xgsn\" (UniqueName: \"kubernetes.io/projected/09ea8db5-8a84-4f64-a520-8089c8ee8231-kube-api-access-2xgsn\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462100    6777 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462111    6777 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-main-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462119    6777 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462127    6777 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-cluster-tls-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:14.462268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:14.462135    6777 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09ea8db5-8a84-4f64-a520-8089c8ee8231-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:15.246305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.246267    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.275351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.275323    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:07:15.279514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.279489    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.310778    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311467    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy-web"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311486    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy-web"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311518    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="alertmanager"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311525    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="alertmanager"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311557    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="init-config-reloader"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311565    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="init-config-reloader"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311584    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="config-reloader"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311593    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="config-reloader"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311619    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy-metric"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311626    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy-metric"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311635    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311642    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311658    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="prom-label-proxy"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311665    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="prom-label-proxy"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311813    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="init-config-reloader"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311826    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="config-reloader"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311843    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy-metric"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311851    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311865    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="prom-label-proxy"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311875    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="kube-rbac-proxy-web"
Oct 30 19:07:15.313797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.311892    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" containerName="alertmanager"
Oct 30 19:07:15.341849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.341821    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:07:15.342032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.341958    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.348277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.348248    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:07:15.348277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.348248    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:07:15.348466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.348281    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:07:15.348466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.348373    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:07:15.348466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.348398    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:07:15.348466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.348439    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:07:15.348466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.348453    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:07:15.348718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.348561    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:07:15.348718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.348618    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sg4ml\""
Oct 30 19:07:15.350461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.350442    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:07:15.470764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.470720    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-config-volume\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.470764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.470764    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgcv\" (UniqueName: \"kubernetes.io/projected/73033c1c-bdf2-41a5-8a23-8fe025e3df89-kube-api-access-gkgcv\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.470788    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73033c1c-bdf2-41a5-8a23-8fe025e3df89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.470954    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.471026    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73033c1c-bdf2-41a5-8a23-8fe025e3df89-config-out\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.471110    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.471142    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73033c1c-bdf2-41a5-8a23-8fe025e3df89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.471160    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.471202    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.471237    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-web-config\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.471272    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/73033c1c-bdf2-41a5-8a23-8fe025e3df89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.471302    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.471405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.471320    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73033c1c-bdf2-41a5-8a23-8fe025e3df89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.571907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.571866    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.571913    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73033c1c-bdf2-41a5-8a23-8fe025e3df89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572048    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-config-volume\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572119    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgcv\" (UniqueName: \"kubernetes.io/projected/73033c1c-bdf2-41a5-8a23-8fe025e3df89-kube-api-access-gkgcv\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572158    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73033c1c-bdf2-41a5-8a23-8fe025e3df89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572212    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572259    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73033c1c-bdf2-41a5-8a23-8fe025e3df89-config-out\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572291    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572332    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73033c1c-bdf2-41a5-8a23-8fe025e3df89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572367    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572414    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572441    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-web-config\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572477    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/73033c1c-bdf2-41a5-8a23-8fe025e3df89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572712    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73033c1c-bdf2-41a5-8a23-8fe025e3df89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.572826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572758    6777 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:07:15.572944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.572851    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/73033c1c-bdf2-41a5-8a23-8fe025e3df89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.573543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.573470    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73033c1c-bdf2-41a5-8a23-8fe025e3df89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.575577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.575549    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73033c1c-bdf2-41a5-8a23-8fe025e3df89-config-out\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.575849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.575823    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.575932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.575843    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.576133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.576109    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-web-config\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.576232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.576133    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-config-volume\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.576232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.576155    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73033c1c-bdf2-41a5-8a23-8fe025e3df89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.576232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.576205    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.576373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.576356    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.577480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.577463    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73033c1c-bdf2-41a5-8a23-8fe025e3df89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.584711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.584692    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgcv\" (UniqueName: \"kubernetes.io/projected/73033c1c-bdf2-41a5-8a23-8fe025e3df89-kube-api-access-gkgcv\") pod \"alertmanager-main-0\" (UID: \"73033c1c-bdf2-41a5-8a23-8fe025e3df89\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:15.756945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:15.756908    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:07:16.071301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.071271    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:07:16.072498 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:07:16.072470    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73033c1c_bdf2_41a5_8a23_8fe025e3df89.slice/crio-85ed5db047acb78b2419487e9b21e335dba6060e7221d409b39b4bcdd4c2f887 WatchSource:0}: Error finding container 85ed5db047acb78b2419487e9b21e335dba6060e7221d409b39b4bcdd4c2f887: Status 404 returned error can't find the container with id 85ed5db047acb78b2419487e9b21e335dba6060e7221d409b39b4bcdd4c2f887
Oct 30 19:07:16.109629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.109603    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ea8db5-8a84-4f64-a520-8089c8ee8231" path="/var/lib/kubelet/pods/09ea8db5-8a84-4f64-a520-8089c8ee8231/volumes"
Oct 30 19:07:16.251222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.251188    6777 generic.go:356] "Generic (PLEG): container finished" podID="73033c1c-bdf2-41a5-8a23-8fe025e3df89" containerID="90ac6694d1ebc9feafe212c7754e7a699fcb15cec118425607d13402c4068ca6" exitCode=0
Oct 30 19:07:16.251566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.251274    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"73033c1c-bdf2-41a5-8a23-8fe025e3df89","Type":"ContainerDied","Data":"90ac6694d1ebc9feafe212c7754e7a699fcb15cec118425607d13402c4068ca6"}
Oct 30 19:07:16.251566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.251344    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"73033c1c-bdf2-41a5-8a23-8fe025e3df89","Type":"ContainerStarted","Data":"85ed5db047acb78b2419487e9b21e335dba6060e7221d409b39b4bcdd4c2f887"}
Oct 30 19:07:16.831297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.831271    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:07:16.831664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.831597    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="prometheus" containerID="cri-o://7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e" gracePeriod=600
Oct 30 19:07:16.831747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.831660    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="thanos-sidecar" containerID="cri-o://af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b" gracePeriod=600
Oct 30 19:07:16.831747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.831667    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy" containerID="cri-o://43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2" gracePeriod=600
Oct 30 19:07:16.831845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.831698    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="config-reloader" containerID="cri-o://95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c" gracePeriod=600
Oct 30 19:07:16.831845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.831764    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9" gracePeriod=600
Oct 30 19:07:16.832034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:16.831969    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy-web" containerID="cri-o://80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee" gracePeriod=600
Oct 30 19:07:17.134639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.134577    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.259132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259096    6777 generic.go:356] "Generic (PLEG): container finished" podID="9d018801-0a52-45a6-a6c5-94692d11163a" containerID="3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9" exitCode=0
Oct 30 19:07:17.259132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259128    6777 generic.go:356] "Generic (PLEG): container finished" podID="9d018801-0a52-45a6-a6c5-94692d11163a" containerID="43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2" exitCode=0
Oct 30 19:07:17.259132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259138    6777 generic.go:356] "Generic (PLEG): container finished" podID="9d018801-0a52-45a6-a6c5-94692d11163a" containerID="80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee" exitCode=0
Oct 30 19:07:17.259132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259150    6777 generic.go:356] "Generic (PLEG): container finished" podID="9d018801-0a52-45a6-a6c5-94692d11163a" containerID="af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b" exitCode=0
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259159    6777 generic.go:356] "Generic (PLEG): container finished" podID="9d018801-0a52-45a6-a6c5-94692d11163a" containerID="95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c" exitCode=0
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259169    6777 generic.go:356] "Generic (PLEG): container finished" podID="9d018801-0a52-45a6-a6c5-94692d11163a" containerID="7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e" exitCode=0
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259175    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259175    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerDied","Data":"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"}
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259246    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerDied","Data":"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"}
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259275    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerDied","Data":"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"}
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259302    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerDied","Data":"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"}
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259328    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerDied","Data":"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"}
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259368    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerDied","Data":"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"}
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259328    6777 scope.go:122] "RemoveContainer" containerID="3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"
Oct 30 19:07:17.259698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.259412    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d018801-0a52-45a6-a6c5-94692d11163a","Type":"ContainerDied","Data":"9bbc37c289521525846e00af8ef2e919394bf5f1a73e51c791777243efefbbcc"}
Oct 30 19:07:17.262961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.262936    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"73033c1c-bdf2-41a5-8a23-8fe025e3df89","Type":"ContainerStarted","Data":"6e61fbd673d5540a5fba0364c28159fbf5b002430ca9dd9d71d7ab1e81bd9fb7"}
Oct 30 19:07:17.263097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.262993    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"73033c1c-bdf2-41a5-8a23-8fe025e3df89","Type":"ContainerStarted","Data":"b5736b9caac5fe7107054701e8089b5cc456b4345f3f9e2b15c673989e9f6f90"}
Oct 30 19:07:17.263097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.263037    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"73033c1c-bdf2-41a5-8a23-8fe025e3df89","Type":"ContainerStarted","Data":"4aa379efb8d8d8beecc52a79d56328bcc0446302be995b2fc785166b528e7e3a"}
Oct 30 19:07:17.263097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.263081    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"73033c1c-bdf2-41a5-8a23-8fe025e3df89","Type":"ContainerStarted","Data":"140771d3bad5587b150a8a2cab85336857afd01e2f78d1bdf349c8ff64f87026"}
Oct 30 19:07:17.263202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.263106    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"73033c1c-bdf2-41a5-8a23-8fe025e3df89","Type":"ContainerStarted","Data":"4e3f1a1aeaaa879da2acb747dba1ff9dc4e971818299d2470cee4b7f95be80c8"}
Oct 30 19:07:17.263202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.263129    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"73033c1c-bdf2-41a5-8a23-8fe025e3df89","Type":"ContainerStarted","Data":"6d1d7363d5bd8209a31396849e7591b64cd8c2e93f41adb97371ec7a97c68299"}
Oct 30 19:07:17.268119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.268099    6777 scope.go:122] "RemoveContainer" containerID="43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"
Oct 30 19:07:17.275102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.275079    6777 scope.go:122] "RemoveContainer" containerID="80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"
Oct 30 19:07:17.281582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.281566    6777 scope.go:122] "RemoveContainer" containerID="af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"
Oct 30 19:07:17.286601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.286582    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.286694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.286621    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.286694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.286651    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.286794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.286701    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.286794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.286731    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.286958    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287042    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287097    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287116    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287160    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287165    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287194    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287225    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287258    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287285    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287314    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287483    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:17.289053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.288236    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:17.289872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.289147    6777 scope.go:122] "RemoveContainer" containerID="95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"
Oct 30 19:07:17.289872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.289245    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:17.289872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.289334    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:17.289872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.289530    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:17.289872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.289758    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:17.289872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.289835    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:17.289872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.287380    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.290583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.289915    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.290583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.289952    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.290583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.289980    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.290583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.290343    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.290583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.290383    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca\") pod \"9d018801-0a52-45a6-a6c5-94692d11163a\" (UID: \"9d018801-0a52-45a6-a6c5-94692d11163a\") "
Oct 30 19:07:17.290583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.290403    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:17.290901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.290727    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:17.290966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.290927    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:17.291374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291271    6777 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291294    6777 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-metrics-client-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291311    6777 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291329    6777 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-kube-rbac-proxy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291345    6777 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291361    6777 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291377    6777 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291393    6777 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291411    6777 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291429    6777 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-db\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291442    6777 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d018801-0a52-45a6-a6c5-94692d11163a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291458    6777 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-metrics-client-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.291990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.291907    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:17.292134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.292107    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:17.292691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.292668    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "kube-api-access-2bnw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:17.293247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.293225    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:17.293334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.293250    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:17.293400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.293329    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:17.294632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.294600    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:17.298439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.298281    6777 scope.go:122] "RemoveContainer" containerID="7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"
Oct 30 19:07:17.298439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.298394    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.291462538 podStartE2EDuration="2.298378332s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.915794ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:17.297526202 +0000 UTC m=+19.294441586" watchObservedRunningTime="2026-10-30 19:07:17.298378332 +0000 UTC m=+19.295293715"
Oct 30 19:07:17.303527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.303501    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config" pod "9d018801-0a52-45a6-a6c5-94692d11163a" (UID: "9d018801-0a52-45a6-a6c5-94692d11163a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:17.305785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.305768    6777 scope.go:122] "RemoveContainer" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"
Oct 30 19:07:17.316084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.316041    6777 scope.go:122] "RemoveContainer" containerID="3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"
Oct 30 19:07:17.316697 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:17.316666    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": container with ID starting with 3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9 not found: ID does not exist" containerID="3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"
Oct 30 19:07:17.316810 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.316707    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"} err="failed to get container status \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": rpc error: code = NotFound desc = could not find container \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": container with ID starting with 3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9 not found: ID does not exist"
Oct 30 19:07:17.316810 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.316745    6777 scope.go:122] "RemoveContainer" containerID="43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"
Oct 30 19:07:17.317051 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:17.317022    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": container with ID starting with 43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2 not found: ID does not exist" containerID="43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"
Oct 30 19:07:17.317051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.317045    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"} err="failed to get container status \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": rpc error: code = NotFound desc = could not find container \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": container with ID starting with 43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2 not found: ID does not exist"
Oct 30 19:07:17.317051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.317084    6777 scope.go:122] "RemoveContainer" containerID="80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"
Oct 30 19:07:17.317051 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:17.317350    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": container with ID starting with 80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee not found: ID does not exist" containerID="80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"
Oct 30 19:07:17.317051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.317370    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"} err="failed to get container status \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": rpc error: code = NotFound desc = could not find container \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": container with ID starting with 80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee not found: ID does not exist"
Oct 30 19:07:17.317051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.317384    6777 scope.go:122] "RemoveContainer" containerID="af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"
Oct 30 19:07:17.317733 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:17.317610    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": container with ID starting with af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b not found: ID does not exist" containerID="af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"
Oct 30 19:07:17.317733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.317638    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"} err="failed to get container status \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": rpc error: code = NotFound desc = could not find container \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": container with ID starting with af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b not found: ID does not exist"
Oct 30 19:07:17.317733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.317660    6777 scope.go:122] "RemoveContainer" containerID="95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"
Oct 30 19:07:17.317927 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:17.317904    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": container with ID starting with 95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c not found: ID does not exist" containerID="95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"
Oct 30 19:07:17.317980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.317931    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"} err="failed to get container status \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": rpc error: code = NotFound desc = could not find container \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": container with ID starting with 95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c not found: ID does not exist"
Oct 30 19:07:17.317980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.317948    6777 scope.go:122] "RemoveContainer" containerID="7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"
Oct 30 19:07:17.318200 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:17.318181    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": container with ID starting with 7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e not found: ID does not exist" containerID="7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"
Oct 30 19:07:17.318272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.318204    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"} err="failed to get container status \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": rpc error: code = NotFound desc = could not find container \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": container with ID starting with 7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e not found: ID does not exist"
Oct 30 19:07:17.318272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.318222    6777 scope.go:122] "RemoveContainer" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"
Oct 30 19:07:17.318490 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:17.318471    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": container with ID starting with 2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02 not found: ID does not exist" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"
Oct 30 19:07:17.318554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.318494    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"} err="failed to get container status \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": rpc error: code = NotFound desc = could not find container \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": container with ID starting with 2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02 not found: ID does not exist"
Oct 30 19:07:17.318554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.318509    6777 scope.go:122] "RemoveContainer" containerID="3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"
Oct 30 19:07:17.318744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.318723    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"} err="failed to get container status \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": rpc error: code = NotFound desc = could not find container \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": container with ID starting with 3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9 not found: ID does not exist"
Oct 30 19:07:17.318816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.318744    6777 scope.go:122] "RemoveContainer" containerID="43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"
Oct 30 19:07:17.318982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.318955    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"} err="failed to get container status \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": rpc error: code = NotFound desc = could not find container \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": container with ID starting with 43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2 not found: ID does not exist"
Oct 30 19:07:17.318982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.318981    6777 scope.go:122] "RemoveContainer" containerID="80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"
Oct 30 19:07:17.319242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.319222    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"} err="failed to get container status \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": rpc error: code = NotFound desc = could not find container \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": container with ID starting with 80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee not found: ID does not exist"
Oct 30 19:07:17.319242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.319241    6777 scope.go:122] "RemoveContainer" containerID="af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"
Oct 30 19:07:17.319454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.319436    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"} err="failed to get container status \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": rpc error: code = NotFound desc = could not find container \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": container with ID starting with af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b not found: ID does not exist"
Oct 30 19:07:17.319454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.319453    6777 scope.go:122] "RemoveContainer" containerID="95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"
Oct 30 19:07:17.319653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.319635    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"} err="failed to get container status \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": rpc error: code = NotFound desc = could not find container \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": container with ID starting with 95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c not found: ID does not exist"
Oct 30 19:07:17.319714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.319653    6777 scope.go:122] "RemoveContainer" containerID="7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"
Oct 30 19:07:17.319894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.319876    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"} err="failed to get container status \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": rpc error: code = NotFound desc = could not find container \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": container with ID starting with 7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e not found: ID does not exist"
Oct 30 19:07:17.319894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.319894    6777 scope.go:122] "RemoveContainer" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"
Oct 30 19:07:17.320150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.320130    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"} err="failed to get container status \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": rpc error: code = NotFound desc = could not find container \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": container with ID starting with 2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02 not found: ID does not exist"
Oct 30 19:07:17.320150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.320149    6777 scope.go:122] "RemoveContainer" containerID="3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"
Oct 30 19:07:17.320404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.320385    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"} err="failed to get container status \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": rpc error: code = NotFound desc = could not find container \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": container with ID starting with 3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9 not found: ID does not exist"
Oct 30 19:07:17.320404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.320403    6777 scope.go:122] "RemoveContainer" containerID="43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"
Oct 30 19:07:17.320622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.320602    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"} err="failed to get container status \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": rpc error: code = NotFound desc = could not find container \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": container with ID starting with 43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2 not found: ID does not exist"
Oct 30 19:07:17.320622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.320621    6777 scope.go:122] "RemoveContainer" containerID="80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"
Oct 30 19:07:17.320865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.320846    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"} err="failed to get container status \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": rpc error: code = NotFound desc = could not find container \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": container with ID starting with 80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee not found: ID does not exist"
Oct 30 19:07:17.320865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.320864    6777 scope.go:122] "RemoveContainer" containerID="af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"
Oct 30 19:07:17.321098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.321079    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"} err="failed to get container status \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": rpc error: code = NotFound desc = could not find container \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": container with ID starting with af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b not found: ID does not exist"
Oct 30 19:07:17.321098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.321097    6777 scope.go:122] "RemoveContainer" containerID="95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"
Oct 30 19:07:17.321323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.321307    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"} err="failed to get container status \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": rpc error: code = NotFound desc = could not find container \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": container with ID starting with 95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c not found: ID does not exist"
Oct 30 19:07:17.321323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.321321    6777 scope.go:122] "RemoveContainer" containerID="7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"
Oct 30 19:07:17.321521 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.321505    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"} err="failed to get container status \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": rpc error: code = NotFound desc = could not find container \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": container with ID starting with 7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e not found: ID does not exist"
Oct 30 19:07:17.321521 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.321520    6777 scope.go:122] "RemoveContainer" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"
Oct 30 19:07:17.321750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.321730    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"} err="failed to get container status \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": rpc error: code = NotFound desc = could not find container \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": container with ID starting with 2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02 not found: ID does not exist"
Oct 30 19:07:17.321750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.321749    6777 scope.go:122] "RemoveContainer" containerID="3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"
Oct 30 19:07:17.322007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.321990    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"} err="failed to get container status \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": rpc error: code = NotFound desc = could not find container \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": container with ID starting with 3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9 not found: ID does not exist"
Oct 30 19:07:17.322007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.322006    6777 scope.go:122] "RemoveContainer" containerID="43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"
Oct 30 19:07:17.322253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.322233    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"} err="failed to get container status \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": rpc error: code = NotFound desc = could not find container \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": container with ID starting with 43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2 not found: ID does not exist"
Oct 30 19:07:17.322253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.322253    6777 scope.go:122] "RemoveContainer" containerID="80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"
Oct 30 19:07:17.322468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.322438    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"} err="failed to get container status \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": rpc error: code = NotFound desc = could not find container \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": container with ID starting with 80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee not found: ID does not exist"
Oct 30 19:07:17.322552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.322461    6777 scope.go:122] "RemoveContainer" containerID="af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"
Oct 30 19:07:17.322724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.322703    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"} err="failed to get container status \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": rpc error: code = NotFound desc = could not find container \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": container with ID starting with af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b not found: ID does not exist"
Oct 30 19:07:17.322724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.322721    6777 scope.go:122] "RemoveContainer" containerID="95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"
Oct 30 19:07:17.322964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.322949    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"} err="failed to get container status \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": rpc error: code = NotFound desc = could not find container \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": container with ID starting with 95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c not found: ID does not exist"
Oct 30 19:07:17.322964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.322963    6777 scope.go:122] "RemoveContainer" containerID="7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"
Oct 30 19:07:17.323253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.323231    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"} err="failed to get container status \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": rpc error: code = NotFound desc = could not find container \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": container with ID starting with 7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e not found: ID does not exist"
Oct 30 19:07:17.323253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.323252    6777 scope.go:122] "RemoveContainer" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"
Oct 30 19:07:17.323482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.323465    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"} err="failed to get container status \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": rpc error: code = NotFound desc = could not find container \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": container with ID starting with 2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02 not found: ID does not exist"
Oct 30 19:07:17.323482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.323482    6777 scope.go:122] "RemoveContainer" containerID="3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"
Oct 30 19:07:17.323764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.323738    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"} err="failed to get container status \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": rpc error: code = NotFound desc = could not find container \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": container with ID starting with 3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9 not found: ID does not exist"
Oct 30 19:07:17.323843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.323765    6777 scope.go:122] "RemoveContainer" containerID="43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"
Oct 30 19:07:17.324045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.324025    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"} err="failed to get container status \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": rpc error: code = NotFound desc = could not find container \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": container with ID starting with 43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2 not found: ID does not exist"
Oct 30 19:07:17.324045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.324045    6777 scope.go:122] "RemoveContainer" containerID="80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"
Oct 30 19:07:17.324346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.324318    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"} err="failed to get container status \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": rpc error: code = NotFound desc = could not find container \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": container with ID starting with 80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee not found: ID does not exist"
Oct 30 19:07:17.324408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.324347    6777 scope.go:122] "RemoveContainer" containerID="af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"
Oct 30 19:07:17.324556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.324534    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"} err="failed to get container status \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": rpc error: code = NotFound desc = could not find container \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": container with ID starting with af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b not found: ID does not exist"
Oct 30 19:07:17.324624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.324556    6777 scope.go:122] "RemoveContainer" containerID="95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"
Oct 30 19:07:17.324796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.324772    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"} err="failed to get container status \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": rpc error: code = NotFound desc = could not find container \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": container with ID starting with 95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c not found: ID does not exist"
Oct 30 19:07:17.324796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.324795    6777 scope.go:122] "RemoveContainer" containerID="7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"
Oct 30 19:07:17.325176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.325053    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"} err="failed to get container status \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": rpc error: code = NotFound desc = could not find container \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": container with ID starting with 7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e not found: ID does not exist"
Oct 30 19:07:17.325176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.325177    6777 scope.go:122] "RemoveContainer" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"
Oct 30 19:07:17.325545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.325527    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"} err="failed to get container status \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": rpc error: code = NotFound desc = could not find container \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": container with ID starting with 2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02 not found: ID does not exist"
Oct 30 19:07:17.325596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.325546    6777 scope.go:122] "RemoveContainer" containerID="3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"
Oct 30 19:07:17.325885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.325864    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9"} err="failed to get container status \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": rpc error: code = NotFound desc = could not find container \"3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9\": container with ID starting with 3db09e830ac738af255b0ddc92d4aa0873934f0db9ccf3ebcfbf322c900356c9 not found: ID does not exist"
Oct 30 19:07:17.325885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.325884    6777 scope.go:122] "RemoveContainer" containerID="43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"
Oct 30 19:07:17.326237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.326221    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2"} err="failed to get container status \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": rpc error: code = NotFound desc = could not find container \"43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2\": container with ID starting with 43debc27c9f6f39a8611c00c5cdb2a602c18ed7dcbe4caab48355f23e54720d2 not found: ID does not exist"
Oct 30 19:07:17.326237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.326236    6777 scope.go:122] "RemoveContainer" containerID="80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"
Oct 30 19:07:17.326484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.326467    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee"} err="failed to get container status \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": rpc error: code = NotFound desc = could not find container \"80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee\": container with ID starting with 80a63f769f375f7ae55980dcb4ee47a05b2723ecb820a3093ad5a72d402573ee not found: ID does not exist"
Oct 30 19:07:17.326484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.326483    6777 scope.go:122] "RemoveContainer" containerID="af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"
Oct 30 19:07:17.326792 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.326743    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b"} err="failed to get container status \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": rpc error: code = NotFound desc = could not find container \"af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b\": container with ID starting with af46c6ee19e78b511d795adccf56d96ded5f0f3eb5d000451094c67e27a3407b not found: ID does not exist"
Oct 30 19:07:17.326846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.326793    6777 scope.go:122] "RemoveContainer" containerID="95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"
Oct 30 19:07:17.327091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.327067    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c"} err="failed to get container status \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": rpc error: code = NotFound desc = could not find container \"95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c\": container with ID starting with 95b2228f2b8f13cdbc543cf70e0e196e157ed0fb95e3cdde49d6948a28e2393c not found: ID does not exist"
Oct 30 19:07:17.327144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.327092    6777 scope.go:122] "RemoveContainer" containerID="7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"
Oct 30 19:07:17.327351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.327332    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e"} err="failed to get container status \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": rpc error: code = NotFound desc = could not find container \"7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e\": container with ID starting with 7addeb6cb37276b17273b11809f51d6daea1e0feefca7e00ba0cb28b591ea99e not found: ID does not exist"
Oct 30 19:07:17.327407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.327351    6777 scope.go:122] "RemoveContainer" containerID="2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"
Oct 30 19:07:17.327583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.327566    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02"} err="failed to get container status \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": rpc error: code = NotFound desc = could not find container \"2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02\": container with ID starting with 2123279c06dcc6a1d30e9f2c5a92bcd6f3b7ea5a2e6cb0a94c14725f25ddab02 not found: ID does not exist"
Oct 30 19:07:17.392581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.392502    6777 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d018801-0a52-45a6-a6c5-94692d11163a-config-out\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.392581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.392533    6777 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-web-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.392766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.392616    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bnw4\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-kube-api-access-2bnw4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.392766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.392629    6777 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-grpc-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.392766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.392646    6777 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.392766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.392659    6777 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.392766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.392691    6777 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d018801-0a52-45a6-a6c5-94692d11163a-tls-assets\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.392766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.392710    6777 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d018801-0a52-45a6-a6c5-94692d11163a-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:17.584333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.584297    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:07:17.589000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.588975    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:07:17.624724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.624694    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:07:17.625004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.624992    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy"
Oct 30 19:07:17.625046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625006    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy"
Oct 30 19:07:17.625046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625019    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy-web"
Oct 30 19:07:17.625046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625025    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy-web"
Oct 30 19:07:17.625046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625037    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="prometheus"
Oct 30 19:07:17.625046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625042    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="prometheus"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625067    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="thanos-sidecar"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625072    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="thanos-sidecar"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625079    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="config-reloader"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625083    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="config-reloader"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625091    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="init-config-reloader"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625096    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="init-config-reloader"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625106    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy-thanos"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625111    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy-thanos"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625158    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy-thanos"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625167    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625173    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="kube-rbac-proxy-web"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625179    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="prometheus"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625185    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="thanos-sidecar"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625190    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="config-reloader"
Oct 30 19:07:17.625196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.625196    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" containerName="init-config-reloader"
Oct 30 19:07:17.661108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.661033    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:07:17.661227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.661149    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.667431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667287    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:07:17.667431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667362    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:07:17.667431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667378    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:07:17.667686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667469    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:07:17.667686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667293    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:07:17.667686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667523    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:07:17.667686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667534    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:07:17.667686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667631    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:07:17.667686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667679    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5b3rasg0scipm\""
Oct 30 19:07:17.667988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667694    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:07:17.667988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667747    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:07:17.667988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.667866    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:07:17.668139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.668032    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:07:17.668963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.668709    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-llnck\""
Oct 30 19:07:17.673484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.673464    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:07:17.673484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.673472    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:07:17.796244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796201    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796249    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796329    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796361    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796380    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796437    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796467    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796527    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796550    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796610    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-config\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796632    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlx8\" (UniqueName: \"kubernetes.io/projected/586bce6a-ff5d-4531-935d-6cada4099b7d-kube-api-access-gqlx8\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796646    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796664    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796720    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796817    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/586bce6a-ff5d-4531-935d-6cada4099b7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.796875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796861    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.797024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796895    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.797024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.796922    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.797024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.797000    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.797186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.797040    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/586bce6a-ff5d-4531-935d-6cada4099b7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897567    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897601    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897628    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897651    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897773    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897819    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897851    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897884    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897915    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.897969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897947    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-config\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.897979    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlx8\" (UniqueName: \"kubernetes.io/projected/586bce6a-ff5d-4531-935d-6cada4099b7d-kube-api-access-gqlx8\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898002    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898031    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898082    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898112    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/586bce6a-ff5d-4531-935d-6cada4099b7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898126    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898143    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898158    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898203    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898256    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/586bce6a-ff5d-4531-935d-6cada4099b7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898462    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898571    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.898772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898602    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.899471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.898802    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.900387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.899631    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.900387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.900136    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.900387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.900322    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.901587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.901561    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.901587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.901570    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.901765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.901711    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.901912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.901864    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/586bce6a-ff5d-4531-935d-6cada4099b7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.902181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.902155    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/586bce6a-ff5d-4531-935d-6cada4099b7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.902728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.902668    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.903178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.903155    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.903543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.903510    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.903670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.903653    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.903736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.903687    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-config\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.903951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.903931    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/586bce6a-ff5d-4531-935d-6cada4099b7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.904495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.904474    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/586bce6a-ff5d-4531-935d-6cada4099b7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:17.912779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:17.912732    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlx8\" (UniqueName: \"kubernetes.io/projected/586bce6a-ff5d-4531-935d-6cada4099b7d-kube-api-access-gqlx8\") pod \"prometheus-k8s-0\" (UID: \"586bce6a-ff5d-4531-935d-6cada4099b7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:18.109935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:18.109901    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d018801-0a52-45a6-a6c5-94692d11163a" path="/var/lib/kubelet/pods/9d018801-0a52-45a6-a6c5-94692d11163a/volumes"
Oct 30 19:07:18.118701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:18.118675    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:18.439762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:18.439736    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:07:18.441319 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:07:18.441294    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586bce6a_ff5d_4531_935d_6cada4099b7d.slice/crio-8cd73d2ea17ac585e79e19f54f85e2c3bc6634f2ef39efd8ede52acdd1765650 WatchSource:0}: Error finding container 8cd73d2ea17ac585e79e19f54f85e2c3bc6634f2ef39efd8ede52acdd1765650: Status 404 returned error can't find the container with id 8cd73d2ea17ac585e79e19f54f85e2c3bc6634f2ef39efd8ede52acdd1765650
Oct 30 19:07:19.273032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:19.272995    6777 generic.go:356] "Generic (PLEG): container finished" podID="586bce6a-ff5d-4531-935d-6cada4099b7d" containerID="3feba9aec749e3ab96a1838ee66ae41ca1725a10231fe8618a42aa7ce393b8f6" exitCode=0
Oct 30 19:07:19.273262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:19.273079    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"586bce6a-ff5d-4531-935d-6cada4099b7d","Type":"ContainerDied","Data":"3feba9aec749e3ab96a1838ee66ae41ca1725a10231fe8618a42aa7ce393b8f6"}
Oct 30 19:07:19.273262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:19.273152    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"586bce6a-ff5d-4531-935d-6cada4099b7d","Type":"ContainerStarted","Data":"8cd73d2ea17ac585e79e19f54f85e2c3bc6634f2ef39efd8ede52acdd1765650"}
Oct 30 19:07:20.280651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:20.280618    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"586bce6a-ff5d-4531-935d-6cada4099b7d","Type":"ContainerStarted","Data":"2eadfab0e04e81699024c521dfeff3f2cba7e1ed6c97055af3e5aecdc5aa3977"}
Oct 30 19:07:20.281086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:20.280686    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"586bce6a-ff5d-4531-935d-6cada4099b7d","Type":"ContainerStarted","Data":"b01411c1aafdaa1da7f17bc1d357ca5bc64f873d89cefe879a462879c6e0645f"}
Oct 30 19:07:20.281086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:20.280716    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"586bce6a-ff5d-4531-935d-6cada4099b7d","Type":"ContainerStarted","Data":"d4559113ee5e5d9508a6225a68379fcc55942f4bc893da0bf604f529ba58dd89"}
Oct 30 19:07:20.281086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:20.280750    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"586bce6a-ff5d-4531-935d-6cada4099b7d","Type":"ContainerStarted","Data":"6e02829ec50c3cc775ecc80405756c43c64215259b7e95f7507cfb145c047ef6"}
Oct 30 19:07:20.281086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:20.280775    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"586bce6a-ff5d-4531-935d-6cada4099b7d","Type":"ContainerStarted","Data":"4a7cd51a7113935d062771e8bbf0221681b60324d850c2a38b9dd1590e61a546"}
Oct 30 19:07:20.281086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:20.280798    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"586bce6a-ff5d-4531-935d-6cada4099b7d","Type":"ContainerStarted","Data":"bbfda8c93119fc621a4edd5cb78068b1020db6cb210d4716b39b7cb2c04b42a5"}
Oct 30 19:07:20.319736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:20.319684    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.29747567 podStartE2EDuration="3.319669902s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.194232ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:20.318285274 +0000 UTC m=+22.315200669" watchObservedRunningTime="2026-10-30 19:07:20.319669902 +0000 UTC m=+22.316585284"
Oct 30 19:07:21.867929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.867891    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-549b45f6d9-d6xxd"]
Oct 30 19:07:21.870280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.870264    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:21.881107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.881082    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549b45f6d9-d6xxd"]
Oct 30 19:07:21.936449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.936418    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxn8\" (UniqueName: \"kubernetes.io/projected/5d85d2f1-025a-4854-b9ac-6afee2997405-kube-api-access-gtxn8\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:21.936622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.936459    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-service-ca\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:21.936622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.936548    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-oauth-config\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:21.936622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.936600    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d85d2f1-025a-4854-b9ac-6afee2997405-tmp\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:21.936762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.936637    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-trusted-ca-bundle\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:21.936762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.936678    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-serving-cert\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:21.936762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.936695    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-oauth-serving-cert\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:21.936849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:21.936774    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-console-config\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.037628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.037588    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-console-config\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.037795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.037632    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxn8\" (UniqueName: \"kubernetes.io/projected/5d85d2f1-025a-4854-b9ac-6afee2997405-kube-api-access-gtxn8\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.037795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.037672    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-service-ca\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.037795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.037714    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-oauth-config\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.037795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.037788    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d85d2f1-025a-4854-b9ac-6afee2997405-tmp\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.037997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.037811    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-trusted-ca-bundle\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.037997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.037854    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-serving-cert\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.037997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.037960    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-oauth-serving-cert\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.038383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.038351    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d85d2f1-025a-4854-b9ac-6afee2997405-tmp\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.038492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.038451    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-console-config\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.038561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.038500    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-service-ca\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.038617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.038592    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-oauth-serving-cert\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.038812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.038790    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-trusted-ca-bundle\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.040282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.040255    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-oauth-config\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.040384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.040364    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-serving-cert\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.049991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.049969    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxn8\" (UniqueName: \"kubernetes.io/projected/5d85d2f1-025a-4854-b9ac-6afee2997405-kube-api-access-gtxn8\") pod \"console-549b45f6d9-d6xxd\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") " pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.180562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.180479    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:22.452483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:22.452277    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549b45f6d9-d6xxd"]
Oct 30 19:07:22.455142 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:07:22.455114    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d85d2f1_025a_4854_b9ac_6afee2997405.slice/crio-2454f78a8ea559baf20e7a36a74c697a527aef8d8e813f58492644cc675df214 WatchSource:0}: Error finding container 2454f78a8ea559baf20e7a36a74c697a527aef8d8e813f58492644cc675df214: Status 404 returned error can't find the container with id 2454f78a8ea559baf20e7a36a74c697a527aef8d8e813f58492644cc675df214
Oct 30 19:07:23.119330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:23.119284    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:23.294164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:23.294128    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b45f6d9-d6xxd" event={"ID":"5d85d2f1-025a-4854-b9ac-6afee2997405","Type":"ContainerStarted","Data":"cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b"}
Oct 30 19:07:23.294326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:23.294182    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b45f6d9-d6xxd" event={"ID":"5d85d2f1-025a-4854-b9ac-6afee2997405","Type":"ContainerStarted","Data":"2454f78a8ea559baf20e7a36a74c697a527aef8d8e813f58492644cc675df214"}
Oct 30 19:07:23.317029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:23.316970    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-549b45f6d9-d6xxd" podStartSLOduration=2.316946868 podStartE2EDuration="2.316946868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:23.316029893 +0000 UTC m=+25.312945278" watchObservedRunningTime="2026-10-30 19:07:23.316946868 +0000 UTC m=+25.313862251"
Oct 30 19:07:24.113222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.113181    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f645b69cc-d5dm6" podUID="f2ebd094-601c-462b-a42c-89e9d6a27c2a" containerName="console" containerID="cri-o://9275219a3681eaffee601a87bf8132bab83bae839d06f67733a8cdadc14ec4db" gracePeriod=15
Oct 30 19:07:24.301410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.301378    6777 generic.go:356] "Generic (PLEG): container finished" podID="f2ebd094-601c-462b-a42c-89e9d6a27c2a" containerID="9275219a3681eaffee601a87bf8132bab83bae839d06f67733a8cdadc14ec4db" exitCode=0
Oct 30 19:07:24.301840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.301452    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f645b69cc-d5dm6" event={"ID":"f2ebd094-601c-462b-a42c-89e9d6a27c2a","Type":"ContainerDied","Data":"9275219a3681eaffee601a87bf8132bab83bae839d06f67733a8cdadc14ec4db"}
Oct 30 19:07:24.358096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.358073    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:07:24.457195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457109    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config\") pod \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") "
Oct 30 19:07:24.457195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457163    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert\") pod \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") "
Oct 30 19:07:24.457195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457186    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config\") pod \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") "
Oct 30 19:07:24.457427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457324    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg\" (UniqueName: \"kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg\") pod \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") "
Oct 30 19:07:24.457427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457384    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert\") pod \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") "
Oct 30 19:07:24.457504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457425    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca\") pod \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") "
Oct 30 19:07:24.457554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457500    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp\") pod \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\" (UID: \"f2ebd094-601c-462b-a42c-89e9d6a27c2a\") "
Oct 30 19:07:24.457609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457571    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config" pod "f2ebd094-601c-462b-a42c-89e9d6a27c2a" (UID: "f2ebd094-601c-462b-a42c-89e9d6a27c2a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:24.457843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457822    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp" pod "f2ebd094-601c-462b-a42c-89e9d6a27c2a" (UID: "f2ebd094-601c-462b-a42c-89e9d6a27c2a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:24.457953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457838    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert" pod "f2ebd094-601c-462b-a42c-89e9d6a27c2a" (UID: "f2ebd094-601c-462b-a42c-89e9d6a27c2a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:24.457953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457856    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca" pod "f2ebd094-601c-462b-a42c-89e9d6a27c2a" (UID: "f2ebd094-601c-462b-a42c-89e9d6a27c2a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:24.457953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457942    6777 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:24.458113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457961    6777 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd094-601c-462b-a42c-89e9d6a27c2a-tmp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:24.458113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457976    6777 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:24.458113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.457991    6777 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd094-601c-462b-a42c-89e9d6a27c2a-oauth-serving-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:24.459614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.459590    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert" pod "f2ebd094-601c-462b-a42c-89e9d6a27c2a" (UID: "f2ebd094-601c-462b-a42c-89e9d6a27c2a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:24.459845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.459822    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config" pod "f2ebd094-601c-462b-a42c-89e9d6a27c2a" (UID: "f2ebd094-601c-462b-a42c-89e9d6a27c2a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:24.459892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.459863    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg" pod "f2ebd094-601c-462b-a42c-89e9d6a27c2a" (UID: "f2ebd094-601c-462b-a42c-89e9d6a27c2a"). InnerVolumeSpecName "kube-api-access-xlrmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:24.558433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.558394    6777 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-serving-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:24.558433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.558427    6777 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebd094-601c-462b-a42c-89e9d6a27c2a-console-oauth-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:24.558433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:24.558437    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrmg\" (UniqueName: \"kubernetes.io/projected/f2ebd094-601c-462b-a42c-89e9d6a27c2a-kube-api-access-xlrmg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:25.306722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:25.306686    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f645b69cc-d5dm6"
Oct 30 19:07:25.307222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:25.306723    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f645b69cc-d5dm6" event={"ID":"f2ebd094-601c-462b-a42c-89e9d6a27c2a","Type":"ContainerDied","Data":"93ee2a35bd62881dac05f034f08a6d150907923921c956046f8846c0803ae693"}
Oct 30 19:07:25.307222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:25.306788    6777 scope.go:122] "RemoveContainer" containerID="9275219a3681eaffee601a87bf8132bab83bae839d06f67733a8cdadc14ec4db"
Oct 30 19:07:25.331683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:25.331653    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f645b69cc-d5dm6"]
Oct 30 19:07:25.335777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:25.335754    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f645b69cc-d5dm6"]
Oct 30 19:07:26.109951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:26.109913    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ebd094-601c-462b-a42c-89e9d6a27c2a" path="/var/lib/kubelet/pods/f2ebd094-601c-462b-a42c-89e9d6a27c2a/volumes"
Oct 30 19:07:28.457787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:28.457758    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldfm8"
Oct 30 19:07:32.181331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:32.181294    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:32.181747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:32.181364    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:32.186298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:32.186276    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:32.339311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:32.339279    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:07:32.394037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:32.394002    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6474bf5b46-vc9zh"]
Oct 30 19:07:35.230497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:35.230466    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k4plb"
Oct 30 19:07:43.638619 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:43.638581    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:07:57.413759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.413685    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6474bf5b46-vc9zh" podUID="a61cc09d-5259-4d19-ab98-2cf90862bef2" containerName="console" containerID="cri-o://1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7" gracePeriod=15
Oct 30 19:07:57.648338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.648301    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:07:57.760093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.759987    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert\") pod \"a61cc09d-5259-4d19-ab98-2cf90862bef2\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") "
Oct 30 19:07:57.760093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760036    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert\") pod \"a61cc09d-5259-4d19-ab98-2cf90862bef2\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") "
Oct 30 19:07:57.760093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760054    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config\") pod \"a61cc09d-5259-4d19-ab98-2cf90862bef2\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") "
Oct 30 19:07:57.760093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760096    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config\") pod \"a61cc09d-5259-4d19-ab98-2cf90862bef2\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") "
Oct 30 19:07:57.760441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760332    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp\") pod \"a61cc09d-5259-4d19-ab98-2cf90862bef2\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") "
Oct 30 19:07:57.760441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760430    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca\") pod \"a61cc09d-5259-4d19-ab98-2cf90862bef2\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") "
Oct 30 19:07:57.760545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760440    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config" pod "a61cc09d-5259-4d19-ab98-2cf90862bef2" (UID: "a61cc09d-5259-4d19-ab98-2cf90862bef2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:57.760545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760452    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert" pod "a61cc09d-5259-4d19-ab98-2cf90862bef2" (UID: "a61cc09d-5259-4d19-ab98-2cf90862bef2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:57.760545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760465    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle\") pod \"a61cc09d-5259-4d19-ab98-2cf90862bef2\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") "
Oct 30 19:07:57.760545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760514    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94\" (UniqueName: \"kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94\") pod \"a61cc09d-5259-4d19-ab98-2cf90862bef2\" (UID: \"a61cc09d-5259-4d19-ab98-2cf90862bef2\") "
Oct 30 19:07:57.760743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760638    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp" pod "a61cc09d-5259-4d19-ab98-2cf90862bef2" (UID: "a61cc09d-5259-4d19-ab98-2cf90862bef2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:57.760806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760779    6777 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61cc09d-5259-4d19-ab98-2cf90862bef2-tmp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:57.760806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760800    6777 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-oauth-serving-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:57.760909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760817    6777 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:57.760909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.760865    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle" pod "a61cc09d-5259-4d19-ab98-2cf90862bef2" (UID: "a61cc09d-5259-4d19-ab98-2cf90862bef2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:57.761094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.761047    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca" pod "a61cc09d-5259-4d19-ab98-2cf90862bef2" (UID: "a61cc09d-5259-4d19-ab98-2cf90862bef2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:57.762418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.762391    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert" pod "a61cc09d-5259-4d19-ab98-2cf90862bef2" (UID: "a61cc09d-5259-4d19-ab98-2cf90862bef2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:57.762525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.762484    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94" pod "a61cc09d-5259-4d19-ab98-2cf90862bef2" (UID: "a61cc09d-5259-4d19-ab98-2cf90862bef2"). InnerVolumeSpecName "kube-api-access-rjd94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:57.762525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.762504    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config" pod "a61cc09d-5259-4d19-ab98-2cf90862bef2" (UID: "a61cc09d-5259-4d19-ab98-2cf90862bef2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:57.861452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.861417    6777 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:57.861452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.861448    6777 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61cc09d-5259-4d19-ab98-2cf90862bef2-trusted-ca-bundle\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:57.861452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.861459    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjd94\" (UniqueName: \"kubernetes.io/projected/a61cc09d-5259-4d19-ab98-2cf90862bef2-kube-api-access-rjd94\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:57.861679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.861470    6777 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-serving-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:57.861679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:57.861480    6777 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61cc09d-5259-4d19-ab98-2cf90862bef2-console-oauth-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:58.452876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.452781    6777 generic.go:356] "Generic (PLEG): container finished" podID="a61cc09d-5259-4d19-ab98-2cf90862bef2" containerID="1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7" exitCode=0
Oct 30 19:07:58.452876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.452842    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6474bf5b46-vc9zh"
Oct 30 19:07:58.452876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.452868    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6474bf5b46-vc9zh" event={"ID":"a61cc09d-5259-4d19-ab98-2cf90862bef2","Type":"ContainerDied","Data":"1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7"}
Oct 30 19:07:58.453445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.452916    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6474bf5b46-vc9zh" event={"ID":"a61cc09d-5259-4d19-ab98-2cf90862bef2","Type":"ContainerDied","Data":"1b797ba4614efd885de78c116712da8e7c1ea06d216d954dbf2978674be28bea"}
Oct 30 19:07:58.453445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.452937    6777 scope.go:122] "RemoveContainer" containerID="1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7"
Oct 30 19:07:58.461936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.461918    6777 scope.go:122] "RemoveContainer" containerID="1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7"
Oct 30 19:07:58.462386 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:07:58.462318    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7\": container with ID starting with 1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7 not found: ID does not exist" containerID="1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7"
Oct 30 19:07:58.462386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.462352    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7"} err="failed to get container status \"1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7\": rpc error: code = NotFound desc = could not find container \"1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7\": container with ID starting with 1232b887e61e2281ad14e9d4feb9fce7748c686bb24f30cfe0defbb841482ba7 not found: ID does not exist"
Oct 30 19:07:58.476892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.476861    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6474bf5b46-vc9zh"]
Oct 30 19:07:58.477401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.477380    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6474bf5b46-vc9zh"]
Oct 30 19:07:58.622101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.622043    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:58.637784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:58.637755    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:07:59.473150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:07:59.473123    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:08:00.110265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:00.110231    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a61cc09d-5259-4d19-ab98-2cf90862bef2" path="/var/lib/kubelet/pods/a61cc09d-5259-4d19-ab98-2cf90862bef2/volumes"
Oct 30 19:08:13.637348 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:08:13.637308    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:08:17.902074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.902032    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8466c667cd-xjs89"]
Oct 30 19:08:17.902707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.902566    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f2ebd094-601c-462b-a42c-89e9d6a27c2a" containerName="console"
Oct 30 19:08:17.902707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.902587    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ebd094-601c-462b-a42c-89e9d6a27c2a" containerName="console"
Oct 30 19:08:17.902707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.902622    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a61cc09d-5259-4d19-ab98-2cf90862bef2" containerName="console"
Oct 30 19:08:17.902707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.902629    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61cc09d-5259-4d19-ab98-2cf90862bef2" containerName="console"
Oct 30 19:08:17.902707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.902702    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61cc09d-5259-4d19-ab98-2cf90862bef2" containerName="console"
Oct 30 19:08:17.902958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.902719    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ebd094-601c-462b-a42c-89e9d6a27c2a" containerName="console"
Oct 30 19:08:17.904886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.904862    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:17.917482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.917452    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8466c667cd-xjs89"]
Oct 30 19:08:17.929658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.929624    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-oauth-serving-cert\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:17.929820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.929684    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13941430-a959-495b-8ccc-9457c5de5236-console-oauth-config\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:17.929820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.929774    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnsh\" (UniqueName: \"kubernetes.io/projected/13941430-a959-495b-8ccc-9457c5de5236-kube-api-access-jtnsh\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:17.929820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.929807    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-service-ca\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:17.930047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.929840    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-trusted-ca-bundle\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:17.930047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.929973    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-console-config\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:17.930047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.930010    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13941430-a959-495b-8ccc-9457c5de5236-console-serving-cert\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:17.930191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:17.930123    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13941430-a959-495b-8ccc-9457c5de5236-tmp\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.031428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.031383    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-oauth-serving-cert\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.031428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.031432    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13941430-a959-495b-8ccc-9457c5de5236-console-oauth-config\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.031659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.031546    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnsh\" (UniqueName: \"kubernetes.io/projected/13941430-a959-495b-8ccc-9457c5de5236-kube-api-access-jtnsh\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.031659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.031591    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-service-ca\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.031761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.031739    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-trusted-ca-bundle\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.031835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.031815    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-console-config\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.032038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.031999    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13941430-a959-495b-8ccc-9457c5de5236-console-serving-cert\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.032173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.032099    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13941430-a959-495b-8ccc-9457c5de5236-tmp\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.032173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.032147    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-oauth-serving-cert\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.032339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.032307    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-service-ca\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.032436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.032419    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13941430-a959-495b-8ccc-9457c5de5236-tmp\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.032539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.032519    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-console-config\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.032775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.032758    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13941430-a959-495b-8ccc-9457c5de5236-trusted-ca-bundle\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.033986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.033967    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13941430-a959-495b-8ccc-9457c5de5236-console-oauth-config\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.034346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.034328    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13941430-a959-495b-8ccc-9457c5de5236-console-serving-cert\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.045036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.044999    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnsh\" (UniqueName: \"kubernetes.io/projected/13941430-a959-495b-8ccc-9457c5de5236-kube-api-access-jtnsh\") pod \"console-8466c667cd-xjs89\" (UID: \"13941430-a959-495b-8ccc-9457c5de5236\") " pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.215656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.215549    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:18.487745 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.487581    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8466c667cd-xjs89"]
Oct 30 19:08:18.490482 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:08:18.490454    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13941430_a959_495b_8ccc_9457c5de5236.slice/crio-0aefecf08b183e2072ea2a07ee6aa42d2a596609729ee6ee3e291ce6dcdfe425 WatchSource:0}: Error finding container 0aefecf08b183e2072ea2a07ee6aa42d2a596609729ee6ee3e291ce6dcdfe425: Status 404 returned error can't find the container with id 0aefecf08b183e2072ea2a07ee6aa42d2a596609729ee6ee3e291ce6dcdfe425
Oct 30 19:08:18.531684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:18.531655    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8466c667cd-xjs89" event={"ID":"13941430-a959-495b-8ccc-9457c5de5236","Type":"ContainerStarted","Data":"0aefecf08b183e2072ea2a07ee6aa42d2a596609729ee6ee3e291ce6dcdfe425"}
Oct 30 19:08:19.540966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:19.540927    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8466c667cd-xjs89" event={"ID":"13941430-a959-495b-8ccc-9457c5de5236","Type":"ContainerStarted","Data":"205b5125562aa6f552a0c0a93b0c1c34fba6d04571cfbc40cfd706b97e835311"}
Oct 30 19:08:19.564301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:19.564250    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8466c667cd-xjs89" podStartSLOduration=2.564237659 podStartE2EDuration="2.564237659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:19.562596926 +0000 UTC m=+81.559512320" watchObservedRunningTime="2026-10-30 19:08:19.564237659 +0000 UTC m=+81.561153044"
Oct 30 19:08:28.216352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:28.216290    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:28.216352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:28.216369    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:28.221200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:28.221172    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:28.579112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:28.579076    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8466c667cd-xjs89"
Oct 30 19:08:28.635015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:28.634985    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549b45f6d9-d6xxd"]
Oct 30 19:08:43.638710 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:08:43.638624    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:08:53.654347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.654285    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-549b45f6d9-d6xxd" podUID="5d85d2f1-025a-4854-b9ac-6afee2997405" containerName="console" containerID="cri-o://cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b" gracePeriod=15
Oct 30 19:08:53.884894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.884868    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:08:53.949558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.949465    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-console-config\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-console-config\") pod \"5d85d2f1-025a-4854-b9ac-6afee2997405\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") "
Oct 30 19:08:53.949558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.949522    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d85d2f1-025a-4854-b9ac-6afee2997405-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d85d2f1-025a-4854-b9ac-6afee2997405-tmp\") pod \"5d85d2f1-025a-4854-b9ac-6afee2997405\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") "
Oct 30 19:08:53.949763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.949568    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-oauth-serving-cert\") pod \"5d85d2f1-025a-4854-b9ac-6afee2997405\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") "
Oct 30 19:08:53.949763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.949604    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d85d2f1-025a-4854-b9ac-6afee2997405-kube-api-access-gtxn8\" (UniqueName: \"kubernetes.io/projected/5d85d2f1-025a-4854-b9ac-6afee2997405-kube-api-access-gtxn8\") pod \"5d85d2f1-025a-4854-b9ac-6afee2997405\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") "
Oct 30 19:08:53.949763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.949638    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-trusted-ca-bundle\") pod \"5d85d2f1-025a-4854-b9ac-6afee2997405\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") "
Oct 30 19:08:53.949763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.949716    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-serving-cert\") pod \"5d85d2f1-025a-4854-b9ac-6afee2997405\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") "
Oct 30 19:08:53.949763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.949751    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-service-ca\") pod \"5d85d2f1-025a-4854-b9ac-6afee2997405\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") "
Oct 30 19:08:53.949970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.949775    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-oauth-config\") pod \"5d85d2f1-025a-4854-b9ac-6afee2997405\" (UID: \"5d85d2f1-025a-4854-b9ac-6afee2997405\") "
Oct 30 19:08:53.949970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.949859    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d85d2f1-025a-4854-b9ac-6afee2997405-tmp" pod "5d85d2f1-025a-4854-b9ac-6afee2997405" (UID: "5d85d2f1-025a-4854-b9ac-6afee2997405"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:53.950099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.950038    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-console-config" pod "5d85d2f1-025a-4854-b9ac-6afee2997405" (UID: "5d85d2f1-025a-4854-b9ac-6afee2997405"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:53.950165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.950117    6777 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d85d2f1-025a-4854-b9ac-6afee2997405-tmp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:53.950225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.950203    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-oauth-serving-cert" pod "5d85d2f1-025a-4854-b9ac-6afee2997405" (UID: "5d85d2f1-025a-4854-b9ac-6afee2997405"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:53.950308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.950262    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-service-ca" pod "5d85d2f1-025a-4854-b9ac-6afee2997405" (UID: "5d85d2f1-025a-4854-b9ac-6afee2997405"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:53.950379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.950298    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-trusted-ca-bundle" pod "5d85d2f1-025a-4854-b9ac-6afee2997405" (UID: "5d85d2f1-025a-4854-b9ac-6afee2997405"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:53.951852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.951828    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d85d2f1-025a-4854-b9ac-6afee2997405-kube-api-access-gtxn8" pod "5d85d2f1-025a-4854-b9ac-6afee2997405" (UID: "5d85d2f1-025a-4854-b9ac-6afee2997405"). InnerVolumeSpecName "kube-api-access-gtxn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:53.951922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.951891    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-serving-cert" pod "5d85d2f1-025a-4854-b9ac-6afee2997405" (UID: "5d85d2f1-025a-4854-b9ac-6afee2997405"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:53.952010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:53.951993    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-oauth-config" pod "5d85d2f1-025a-4854-b9ac-6afee2997405" (UID: "5d85d2f1-025a-4854-b9ac-6afee2997405"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:54.050955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.050921    6777 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-console-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:54.050955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.050954    6777 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-oauth-serving-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:54.050955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.050964    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtxn8\" (UniqueName: \"kubernetes.io/projected/5d85d2f1-025a-4854-b9ac-6afee2997405-kube-api-access-gtxn8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:54.051204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.050975    6777 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-trusted-ca-bundle\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:54.051204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.050984    6777 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-serving-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:54.051204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.050993    6777 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d85d2f1-025a-4854-b9ac-6afee2997405-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:54.051204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.051003    6777 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d85d2f1-025a-4854-b9ac-6afee2997405-console-oauth-config\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:54.675171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.675131    6777 generic.go:356] "Generic (PLEG): container finished" podID="5d85d2f1-025a-4854-b9ac-6afee2997405" containerID="cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b" exitCode=0
Oct 30 19:08:54.675604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.675188    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b45f6d9-d6xxd"
Oct 30 19:08:54.675604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.675218    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b45f6d9-d6xxd" event={"ID":"5d85d2f1-025a-4854-b9ac-6afee2997405","Type":"ContainerDied","Data":"cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b"}
Oct 30 19:08:54.675604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.675287    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b45f6d9-d6xxd" event={"ID":"5d85d2f1-025a-4854-b9ac-6afee2997405","Type":"ContainerDied","Data":"2454f78a8ea559baf20e7a36a74c697a527aef8d8e813f58492644cc675df214"}
Oct 30 19:08:54.675604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.675311    6777 scope.go:122] "RemoveContainer" containerID="cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b"
Oct 30 19:08:54.683629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.683605    6777 scope.go:122] "RemoveContainer" containerID="cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b"
Oct 30 19:08:54.683914 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:08:54.683893    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b\": container with ID starting with cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b not found: ID does not exist" containerID="cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b"
Oct 30 19:08:54.683967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.683922    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b"} err="failed to get container status \"cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b\": rpc error: code = NotFound desc = could not find container \"cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b\": container with ID starting with cf144a202f5b9cda72268359c7cb35330c6fa29126edc0d227884eb613be762b not found: ID does not exist"
Oct 30 19:08:54.692656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.692632    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549b45f6d9-d6xxd"]
Oct 30 19:08:54.699225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:54.699204    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-549b45f6d9-d6xxd"]
Oct 30 19:08:56.114791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:08:56.114757    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d85d2f1-025a-4854-b9ac-6afee2997405" path="/var/lib/kubelet/pods/5d85d2f1-025a-4854-b9ac-6afee2997405/volumes"
Oct 30 19:09:13.638820 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:09:13.638779    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:09:43.638267 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:09:43.638224    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:10:13.638739 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:10:13.638652    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:10:43.638497 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:10:43.638458    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:10:59.156421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.156377    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-hb9mz"]
Oct 30 19:10:59.158674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.156749    6777 cpu_manager.go:414] "containerMap: removing container" podUID="5d85d2f1-025a-4854-b9ac-6afee2997405" containerName="console"
Oct 30 19:10:59.158674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.156761    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d85d2f1-025a-4854-b9ac-6afee2997405" containerName="console"
Oct 30 19:10:59.158674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.156812    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d85d2f1-025a-4854-b9ac-6afee2997405" containerName="console"
Oct 30 19:10:59.159534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.159519    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hb9mz"
Oct 30 19:10:59.164963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.164937    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 19:10:59.165115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.164982    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 19:10:59.165115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.165005    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pzncc\""
Oct 30 19:10:59.170368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.170339    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hb9mz"]
Oct 30 19:10:59.249747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.249710    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsf6n\" (UniqueName: \"kubernetes.io/projected/9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08-kube-api-access-jsf6n\") pod \"cert-manager-597b95cb84-hb9mz\" (UID: \"9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08\") " pod="cert-manager/cert-manager-597b95cb84-hb9mz"
Oct 30 19:10:59.249932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.249761    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08-bound-sa-token\") pod \"cert-manager-597b95cb84-hb9mz\" (UID: \"9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08\") " pod="cert-manager/cert-manager-597b95cb84-hb9mz"
Oct 30 19:10:59.351037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.350993    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsf6n\" (UniqueName: \"kubernetes.io/projected/9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08-kube-api-access-jsf6n\") pod \"cert-manager-597b95cb84-hb9mz\" (UID: \"9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08\") " pod="cert-manager/cert-manager-597b95cb84-hb9mz"
Oct 30 19:10:59.351246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.351079    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08-bound-sa-token\") pod \"cert-manager-597b95cb84-hb9mz\" (UID: \"9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08\") " pod="cert-manager/cert-manager-597b95cb84-hb9mz"
Oct 30 19:10:59.363827 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.363801    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08-bound-sa-token\") pod \"cert-manager-597b95cb84-hb9mz\" (UID: \"9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08\") " pod="cert-manager/cert-manager-597b95cb84-hb9mz"
Oct 30 19:10:59.363965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.363876    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsf6n\" (UniqueName: \"kubernetes.io/projected/9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08-kube-api-access-jsf6n\") pod \"cert-manager-597b95cb84-hb9mz\" (UID: \"9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08\") " pod="cert-manager/cert-manager-597b95cb84-hb9mz"
Oct 30 19:10:59.489417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.489340    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hb9mz"
Oct 30 19:10:59.763034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.762809    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hb9mz"]
Oct 30 19:10:59.765851 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:10:59.765825    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9aef91f6_b5a4_4b2c_a0ea_7bb2d8314b08.slice/crio-0a5c11ac3deb69bfbd41627e4724e20eb267ac1ba6cd1d2dd8a89b1b630267eb WatchSource:0}: Error finding container 0a5c11ac3deb69bfbd41627e4724e20eb267ac1ba6cd1d2dd8a89b1b630267eb: Status 404 returned error can't find the container with id 0a5c11ac3deb69bfbd41627e4724e20eb267ac1ba6cd1d2dd8a89b1b630267eb
Oct 30 19:10:59.767763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:10:59.767747    6777 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:11:00.147250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:00.147214    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hb9mz" event={"ID":"9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08","Type":"ContainerStarted","Data":"0a5c11ac3deb69bfbd41627e4724e20eb267ac1ba6cd1d2dd8a89b1b630267eb"}
Oct 30 19:11:03.168027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:03.167916    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hb9mz" event={"ID":"9aef91f6-b5a4-4b2c-a0ea-7bb2d8314b08","Type":"ContainerStarted","Data":"e9ea94a9370b166ecf189a86aa8ecd24c31eb09943106ae1dd92d1b5a6ff840e"}
Oct 30 19:11:03.186922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:03.186868    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-hb9mz" podStartSLOduration=1.046896616 podStartE2EDuration="4.186855239s" totalImagesPullingTime="3.139958623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:03.186146227 +0000 UTC m=+245.183061628" watchObservedRunningTime="2026-10-30 19:11:03.186855239 +0000 UTC m=+245.183770621"
Oct 30 19:11:13.637647 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:11:13.637607    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:11:25.567142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.567112    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"]
Oct 30 19:11:25.570645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.570622    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.578577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.578555    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 19:11:25.582801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.582783    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:11:25.583112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.583091    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 19:11:25.583173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.583130    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 19:11:25.584203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.584181    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"]
Oct 30 19:11:25.685001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.684972    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-djrzj\""
Oct 30 19:11:25.685188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.685006    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 19:11:25.709429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.709392    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-manager-config\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.709613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.709441    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-metrics-cert\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.709613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.709576    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4qw\" (UniqueName: \"kubernetes.io/projected/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-kube-api-access-nn4qw\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.709734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.709632    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-cert\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.810335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.810301    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-cert\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.810507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.810388    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-manager-config\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.810507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.810459    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-metrics-cert\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.810624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.810511    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4qw\" (UniqueName: \"kubernetes.io/projected/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-kube-api-access-nn4qw\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.811168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.811148    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-manager-config\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.812844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.812815    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-metrics-cert\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.812933 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.812848    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-cert\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.823698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.823643    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4qw\" (UniqueName: \"kubernetes.io/projected/2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2-kube-api-access-nn4qw\") pod \"lws-controller-manager-8c744547b-vztcl\" (UID: \"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:25.882703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:25.882664    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:26.154112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:26.154084    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"]
Oct 30 19:11:26.156819 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:11:26.156792    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca3cd3a_7920_4633_b0e7_3db65a8e5eb2.slice/crio-c6e10b976216fcf9640dfd633c65cf440b5515ad1265782b243683355884d272 WatchSource:0}: Error finding container c6e10b976216fcf9640dfd633c65cf440b5515ad1265782b243683355884d272: Status 404 returned error can't find the container with id c6e10b976216fcf9640dfd633c65cf440b5515ad1265782b243683355884d272
Oct 30 19:11:26.261142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:26.261103    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl" event={"ID":"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2","Type":"ContainerStarted","Data":"c6e10b976216fcf9640dfd633c65cf440b5515ad1265782b243683355884d272"}
Oct 30 19:11:29.274244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:29.274154    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl" event={"ID":"2ca3cd3a-7920-4633-b0e7-3db65a8e5eb2","Type":"ContainerStarted","Data":"f83d9259416d94828248ceb004df684c908999b66e98632a5710b8e8c2a284e5"}
Oct 30 19:11:29.274244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:29.274237    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:29.293819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:29.293758    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl" podStartSLOduration=1.5969571390000001 podStartE2EDuration="4.293741618s" totalImagesPullingTime="2.696784479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:29.293396996 +0000 UTC m=+271.290312381" watchObservedRunningTime="2026-10-30 19:11:29.293741618 +0000 UTC m=+271.290657003"
Oct 30 19:11:37.164368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.164329    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"]
Oct 30 19:11:37.169000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.168982    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.198667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.198627    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"]
Oct 30 19:11:37.248516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.248480    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:11:37.248695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.248647    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:11:37.248695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.248668    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:11:37.248799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.248741    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:11:37.248889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.248874    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:11:37.248983 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.248968    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v5t6d\""
Oct 30 19:11:37.321938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.321903    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35a6edc2-ad9d-4077-b69d-e67245ba689d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.322148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.321944    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqf7\" (UniqueName: \"kubernetes.io/projected/35a6edc2-ad9d-4077-b69d-e67245ba689d-kube-api-access-zlqf7\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.322148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.322049    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35a6edc2-ad9d-4077-b69d-e67245ba689d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.322148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.322130    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35a6edc2-ad9d-4077-b69d-e67245ba689d-tmp\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.322264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.322153    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35a6edc2-ad9d-4077-b69d-e67245ba689d-manifests\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.322264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.322174    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35a6edc2-ad9d-4077-b69d-e67245ba689d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.422833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.422751    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35a6edc2-ad9d-4077-b69d-e67245ba689d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.422833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.422803    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35a6edc2-ad9d-4077-b69d-e67245ba689d-tmp\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.423045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.422892    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35a6edc2-ad9d-4077-b69d-e67245ba689d-manifests\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.423045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.422911    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35a6edc2-ad9d-4077-b69d-e67245ba689d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.423045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.422952    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35a6edc2-ad9d-4077-b69d-e67245ba689d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.423045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.422974    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqf7\" (UniqueName: \"kubernetes.io/projected/35a6edc2-ad9d-4077-b69d-e67245ba689d-kube-api-access-zlqf7\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.423382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.423352    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35a6edc2-ad9d-4077-b69d-e67245ba689d-manifests\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.425342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.425317    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35a6edc2-ad9d-4077-b69d-e67245ba689d-tmp\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.425687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.425664    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35a6edc2-ad9d-4077-b69d-e67245ba689d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.425759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.425685    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35a6edc2-ad9d-4077-b69d-e67245ba689d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.425759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.425695    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35a6edc2-ad9d-4077-b69d-e67245ba689d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.435715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.435695    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqf7\" (UniqueName: \"kubernetes.io/projected/35a6edc2-ad9d-4077-b69d-e67245ba689d-kube-api-access-zlqf7\") pod \"opendatahub-operator-controller-manager-7c94c66984-4t9lk\" (UID: \"35a6edc2-ad9d-4077-b69d-e67245ba689d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.482567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.482533    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:37.798291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:37.798254    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"]
Oct 30 19:11:37.802174 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:11:37.802139    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a6edc2_ad9d_4077_b69d_e67245ba689d.slice/crio-101ddadd6be28a073f467ae3befb7e12830b665c5eabb406de2f54c940d45cdc WatchSource:0}: Error finding container 101ddadd6be28a073f467ae3befb7e12830b665c5eabb406de2f54c940d45cdc: Status 404 returned error can't find the container with id 101ddadd6be28a073f467ae3befb7e12830b665c5eabb406de2f54c940d45cdc
Oct 30 19:11:38.311332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:38.311284    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk" event={"ID":"35a6edc2-ad9d-4077-b69d-e67245ba689d","Type":"ContainerStarted","Data":"101ddadd6be28a073f467ae3befb7e12830b665c5eabb406de2f54c940d45cdc"}
Oct 30 19:11:40.280769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:40.280734    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8c744547b-vztcl"
Oct 30 19:11:41.329423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:41.329382    6777 generic.go:356] "Generic (PLEG): container finished" podID="35a6edc2-ad9d-4077-b69d-e67245ba689d" containerID="e925ec9dd5f2800867920dd5bf319d8d58100b682e84d93ec8986af8281d567f" exitCode=0
Oct 30 19:11:41.329900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:41.329435    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk" event={"ID":"35a6edc2-ad9d-4077-b69d-e67245ba689d","Type":"ContainerDied","Data":"e925ec9dd5f2800867920dd5bf319d8d58100b682e84d93ec8986af8281d567f"}
Oct 30 19:11:42.335388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:42.335345    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk" event={"ID":"35a6edc2-ad9d-4077-b69d-e67245ba689d","Type":"ContainerStarted","Data":"847d1dd4039276671a9d9f5e0134dc92094891412aaa2e4ec362c8d349ea9b73"}
Oct 30 19:11:42.335800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:42.335445    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:42.383377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:42.383318    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk" podStartSLOduration=1.598882519 podStartE2EDuration="5.383302266s" totalImagesPullingTime="3.300983423s" totalInitContainerRuntime="483.436324ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:42.38162291 +0000 UTC m=+284.378538290" watchObservedRunningTime="2026-10-30 19:11:42.383302266 +0000 UTC m=+284.380217695"
Oct 30 19:11:43.639222 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:11:43.639183    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:11:53.341823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:53.341787    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-4t9lk"
Oct 30 19:11:56.041901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.041867    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"]
Oct 30 19:11:56.048443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.048418    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.073952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.073759    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"]
Oct 30 19:11:56.077884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.077852    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e038566-ad43-4e4b-b314-ade25d7b389b-tls-certs\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.078033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.077898    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wx8z\" (UniqueName: \"kubernetes.io/projected/5e038566-ad43-4e4b-b314-ade25d7b389b-kube-api-access-9wx8z\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.078033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.077926    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e038566-ad43-4e4b-b314-ade25d7b389b-tmp\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.130296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.130262    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 19:11:56.130493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.130302    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6zhvc\""
Oct 30 19:11:56.130493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.130272    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 19:11:56.130744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.130723    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:11:56.130899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.130847    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:11:56.179507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.179467    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e038566-ad43-4e4b-b314-ade25d7b389b-tls-certs\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.179507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.179503    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wx8z\" (UniqueName: \"kubernetes.io/projected/5e038566-ad43-4e4b-b314-ade25d7b389b-kube-api-access-9wx8z\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.179721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.179530    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e038566-ad43-4e4b-b314-ade25d7b389b-tmp\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.179721 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:11:56.179639    6777 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Oct 30 19:11:56.179794 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:11:56.179723    6777 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e038566-ad43-4e4b-b314-ade25d7b389b-tls-certs podName:5e038566-ad43-4e4b-b314-ade25d7b389b nodeName:}" failed. No retries permitted until 2026-10-30 19:11:56.679701273 +0000 UTC m=+298.676616654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5e038566-ad43-4e4b-b314-ade25d7b389b-tls-certs") pod "kube-auth-proxy-678bf485d7-bnc5s" (UID: "5e038566-ad43-4e4b-b314-ade25d7b389b") : secret "kube-auth-proxy-tls" not found
Oct 30 19:11:56.181815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.181792    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e038566-ad43-4e4b-b314-ade25d7b389b-tmp\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.200098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.200049    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wx8z\" (UniqueName: \"kubernetes.io/projected/5e038566-ad43-4e4b-b314-ade25d7b389b-kube-api-access-9wx8z\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.684531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.684499    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e038566-ad43-4e4b-b314-ade25d7b389b-tls-certs\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.686876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.686849    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e038566-ad43-4e4b-b314-ade25d7b389b-tls-certs\") pod \"kube-auth-proxy-678bf485d7-bnc5s\" (UID: \"5e038566-ad43-4e4b-b314-ade25d7b389b\") " pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:56.960116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:56.960028    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"
Oct 30 19:11:57.248030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:57.248003    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s"]
Oct 30 19:11:57.250746 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:11:57.250716    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e038566_ad43_4e4b_b314_ade25d7b389b.slice/crio-b14ba102f66f6bf7eb66ffeaac10c23c60645a3da37df9cba00c76814129ffb4 WatchSource:0}: Error finding container b14ba102f66f6bf7eb66ffeaac10c23c60645a3da37df9cba00c76814129ffb4: Status 404 returned error can't find the container with id b14ba102f66f6bf7eb66ffeaac10c23c60645a3da37df9cba00c76814129ffb4
Oct 30 19:11:57.401889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:57.401852    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s" event={"ID":"5e038566-ad43-4e4b-b314-ade25d7b389b","Type":"ContainerStarted","Data":"b14ba102f66f6bf7eb66ffeaac10c23c60645a3da37df9cba00c76814129ffb4"}
Oct 30 19:11:59.566010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:11:59.565977    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 19:12:00.416140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:12:00.416099    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s" event={"ID":"5e038566-ad43-4e4b-b314-ade25d7b389b","Type":"ContainerStarted","Data":"a46bdfa2f53bba9ba8cd86b915268e1a4d8698e17f206d2a6258f95ba004f224"}
Oct 30 19:12:00.436176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:12:00.436121    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-678bf485d7-bnc5s" podStartSLOduration=2.126316959 podStartE2EDuration="4.436106699s" totalImagesPullingTime="2.30978974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:00.435221135 +0000 UTC m=+302.432136521" watchObservedRunningTime="2026-10-30 19:12:00.436106699 +0000 UTC m=+302.433022078"
Oct 30 19:12:13.638462 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:12:13.638427    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:12:43.638587 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:12:43.638544    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:12:58.147779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:12:58.147748    6777 scope.go:122] "RemoveContainer" containerID="d3495f64628493e7d3741f8321cca49762e390ab26564d8df44c3d089a349490"
Oct 30 19:12:58.155665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:12:58.155642    6777 scope.go:122] "RemoveContainer" containerID="63257b9f7a0346d3bf02b9d3cf07e83134f06fb36f0a62233ca557017218890d"
Oct 30 19:12:58.163017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:12:58.162988    6777 scope.go:122] "RemoveContainer" containerID="818d8e2701017f58ea2fd0a9cf8b328205d5db465ccd3af91d5d8fc0afc7f017"
Oct 30 19:12:58.170652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:12:58.170628    6777 scope.go:122] "RemoveContainer" containerID="477c9e680d4f90899fc2330471567ed48624dd5e6a60b5216daab4e08a9a50f4"
Oct 30 19:12:58.178485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:12:58.178464    6777 scope.go:122] "RemoveContainer" containerID="30cff2d19c3b93498e05f0af04a666158378dc198da2853aaae5b934feabb715"
Oct 30 19:12:58.186229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:12:58.186207    6777 scope.go:122] "RemoveContainer" containerID="17dc037d9f19a96e1c66f1097ad74aa4062b5403bb9860cc39b1ae39ac82ac82"
Oct 30 19:12:58.193377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:12:58.193356    6777 scope.go:122] "RemoveContainer" containerID="91696330db95516eba6dc9dd915b57b425c7232ed18bf3910a9597d4e7f42770"
Oct 30 19:13:13.638889 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:13:13.638806    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:13:43.638947 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:13:43.638900    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:14:13.638707 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:14:13.638669    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:14:17.396422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.396380    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"]
Oct 30 19:14:17.400069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.400031    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:17.411853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.411422    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:14:17.411853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.411534    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:14:17.417112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.417087    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"]
Oct 30 19:14:17.418349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.418324    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ph459\""
Oct 30 19:14:17.443204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.443169    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg6xg\" (UniqueName: \"kubernetes.io/projected/a21747c0-3c68-4cd5-ad36-99ab7832fd81-kube-api-access-bg6xg\") pod \"limitador-operator-controller-manager-856dd584c5-mg5p6\" (UID: \"a21747c0-3c68-4cd5-ad36-99ab7832fd81\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:17.543746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.543713    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg6xg\" (UniqueName: \"kubernetes.io/projected/a21747c0-3c68-4cd5-ad36-99ab7832fd81-kube-api-access-bg6xg\") pod \"limitador-operator-controller-manager-856dd584c5-mg5p6\" (UID: \"a21747c0-3c68-4cd5-ad36-99ab7832fd81\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:17.574513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.574475    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg6xg\" (UniqueName: \"kubernetes.io/projected/a21747c0-3c68-4cd5-ad36-99ab7832fd81-kube-api-access-bg6xg\") pod \"limitador-operator-controller-manager-856dd584c5-mg5p6\" (UID: \"a21747c0-3c68-4cd5-ad36-99ab7832fd81\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:17.720132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.720018    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:17.998834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:17.998810    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"]
Oct 30 19:14:18.001776 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:14:18.001745    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21747c0_3c68_4cd5_ad36_99ab7832fd81.slice/crio-d09755cf63089721810e21f8b4e2734f66442f531add7a921f1956e5edb6e7eb WatchSource:0}: Error finding container d09755cf63089721810e21f8b4e2734f66442f531add7a921f1956e5edb6e7eb: Status 404 returned error can't find the container with id d09755cf63089721810e21f8b4e2734f66442f531add7a921f1956e5edb6e7eb
Oct 30 19:14:18.992753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:18.992720    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6" event={"ID":"a21747c0-3c68-4cd5-ad36-99ab7832fd81","Type":"ContainerStarted","Data":"d09755cf63089721810e21f8b4e2734f66442f531add7a921f1956e5edb6e7eb"}
Oct 30 19:14:22.008580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:22.008539    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6" event={"ID":"a21747c0-3c68-4cd5-ad36-99ab7832fd81","Type":"ContainerStarted","Data":"878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4"}
Oct 30 19:14:22.008972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:22.008730    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:22.039898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:22.039847    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6" podStartSLOduration=1.9315222699999999 podStartE2EDuration="5.0398339s" totalImagesPullingTime="3.10831163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:22.038799492 +0000 UTC m=+444.035714902" watchObservedRunningTime="2026-10-30 19:14:22.0398339 +0000 UTC m=+444.036749285"
Oct 30 19:14:33.015786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:33.015751    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:37.224984 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.224950    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"]
Oct 30 19:14:37.225474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.225217    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" containerName="manager" containerID="cri-o://878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4" gracePeriod=2
Oct 30 19:14:37.230993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.230964    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"]
Oct 30 19:14:37.258098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.258050    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"]
Oct 30 19:14:37.258685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.258665    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" containerName="manager"
Oct 30 19:14:37.258685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.258686    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" containerName="manager"
Oct 30 19:14:37.258833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.258791    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" containerName="manager"
Oct 30 19:14:37.262123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.262101    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"
Oct 30 19:14:37.265148 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:14:37.265112    6777 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mg5p6\" is forbidden: User \"system:node:ip-10-0-132-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-19.us-east-2.compute.internal' and this object" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:37.276608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.276581    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"]
Oct 30 19:14:37.430943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.430913    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9dp\" (UniqueName: \"kubernetes.io/projected/cdf5be3b-460b-4e99-afed-5f4fc0b287cb-kube-api-access-9k9dp\") pod \"limitador-operator-controller-manager-856dd584c5-d7p7h\" (UID: \"cdf5be3b-460b-4e99-afed-5f4fc0b287cb\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"
Oct 30 19:14:37.459389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.459361    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:37.462505 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:14:37.462477    6777 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mg5p6\" is forbidden: User \"system:node:ip-10-0-132-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-19.us-east-2.compute.internal' and this object" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:37.532117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.532079    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9dp\" (UniqueName: \"kubernetes.io/projected/cdf5be3b-460b-4e99-afed-5f4fc0b287cb-kube-api-access-9k9dp\") pod \"limitador-operator-controller-manager-856dd584c5-d7p7h\" (UID: \"cdf5be3b-460b-4e99-afed-5f4fc0b287cb\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"
Oct 30 19:14:37.550028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.549945    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9dp\" (UniqueName: \"kubernetes.io/projected/cdf5be3b-460b-4e99-afed-5f4fc0b287cb-kube-api-access-9k9dp\") pod \"limitador-operator-controller-manager-856dd584c5-d7p7h\" (UID: \"cdf5be3b-460b-4e99-afed-5f4fc0b287cb\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"
Oct 30 19:14:37.631932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.631892    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"
Oct 30 19:14:37.633121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.633051    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21747c0-3c68-4cd5-ad36-99ab7832fd81-kube-api-access-bg6xg\" (UniqueName: \"kubernetes.io/projected/a21747c0-3c68-4cd5-ad36-99ab7832fd81-kube-api-access-bg6xg\") pod \"a21747c0-3c68-4cd5-ad36-99ab7832fd81\" (UID: \"a21747c0-3c68-4cd5-ad36-99ab7832fd81\") "
Oct 30 19:14:37.635716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.635686    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21747c0-3c68-4cd5-ad36-99ab7832fd81-kube-api-access-bg6xg" pod "a21747c0-3c68-4cd5-ad36-99ab7832fd81" (UID: "a21747c0-3c68-4cd5-ad36-99ab7832fd81"). InnerVolumeSpecName "kube-api-access-bg6xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:37.734659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.734621    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg6xg\" (UniqueName: \"kubernetes.io/projected/a21747c0-3c68-4cd5-ad36-99ab7832fd81-kube-api-access-bg6xg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:37.922827 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:37.922798    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"]
Oct 30 19:14:37.925671 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:14:37.925642    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf5be3b_460b_4e99_afed_5f4fc0b287cb.slice/crio-d9ecd0cb21b9a488dbf0a95ca2b40094d674a38cb0556be594ce6df75ccd7743 WatchSource:0}: Error finding container d9ecd0cb21b9a488dbf0a95ca2b40094d674a38cb0556be594ce6df75ccd7743: Status 404 returned error can't find the container with id d9ecd0cb21b9a488dbf0a95ca2b40094d674a38cb0556be594ce6df75ccd7743
Oct 30 19:14:38.078544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.078512    6777 generic.go:356] "Generic (PLEG): container finished" podID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" containerID="878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4" exitCode=0
Oct 30 19:14:38.078740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.078558    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:38.078740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.078606    6777 scope.go:122] "RemoveContainer" containerID="878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4"
Oct 30 19:14:38.080542 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.080508    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h" event={"ID":"cdf5be3b-460b-4e99-afed-5f4fc0b287cb","Type":"ContainerStarted","Data":"ce1f7d1464bf62426fc087db2eb9455d26c59a3a0eeaacf4c1e3a4d9863931ad"}
Oct 30 19:14:38.080655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.080561    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h" event={"ID":"cdf5be3b-460b-4e99-afed-5f4fc0b287cb","Type":"ContainerStarted","Data":"d9ecd0cb21b9a488dbf0a95ca2b40094d674a38cb0556be594ce6df75ccd7743"}
Oct 30 19:14:38.080655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.080590    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"
Oct 30 19:14:38.081754 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:14:38.081728    6777 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mg5p6\" is forbidden: User \"system:node:ip-10-0-132-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-19.us-east-2.compute.internal' and this object" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:38.084584 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:14:38.084555    6777 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mg5p6\" is forbidden: User \"system:node:ip-10-0-132-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-19.us-east-2.compute.internal' and this object" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:38.087610 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.087593    6777 scope.go:122] "RemoveContainer" containerID="878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4"
Oct 30 19:14:38.087873 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:14:38.087857    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4\": container with ID starting with 878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4 not found: ID does not exist" containerID="878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4"
Oct 30 19:14:38.087911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.087886    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4"} err="failed to get container status \"878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4\": rpc error: code = NotFound desc = could not find container \"878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4\": container with ID starting with 878a120532a5fefa0140a3bf7588d85b9187dafbbf53c4382032adf136ae73c4 not found: ID does not exist"
Oct 30 19:14:38.106868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.106818    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h" podStartSLOduration=1.106801806 podStartE2EDuration="1.106801806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:38.106237529 +0000 UTC m=+460.103152937" watchObservedRunningTime="2026-10-30 19:14:38.106801806 +0000 UTC m=+460.103717192"
Oct 30 19:14:38.109238 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:14:38.109214    6777 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mg5p6\" is forbidden: User \"system:node:ip-10-0-132-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-19.us-east-2.compute.internal' and this object" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:38.111886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:38.111866    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" path="/var/lib/kubelet/pods/a21747c0-3c68-4cd5-ad36-99ab7832fd81/volumes"
Oct 30 19:14:38.114310 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:14:38.114287    6777 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mg5p6\" is forbidden: User \"system:node:ip-10-0-132-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-19.us-east-2.compute.internal' and this object" podUID="a21747c0-3c68-4cd5-ad36-99ab7832fd81" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mg5p6"
Oct 30 19:14:43.641753 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:14:43.641715    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:14:49.088506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:14:49.088468    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-d7p7h"
Oct 30 19:15:13.638316 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:15:13.638267    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:15:19.027899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.027861    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:15:19.031142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.031118    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:19.043162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.043137    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:15:19.043371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.043352    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zs2nd\""
Oct 30 19:15:19.049090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.049071    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 19:15:19.056431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.056407    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:15:19.100195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.100167    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2bq\" (UniqueName: \"kubernetes.io/projected/77d9131c-32d0-4962-9e0c-dbbcaa8a275a-kube-api-access-kr2bq\") pod \"limitador-limitador-7c9b87d574-lzfhp\" (UID: \"77d9131c-32d0-4962-9e0c-dbbcaa8a275a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:19.100344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.100213    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77d9131c-32d0-4962-9e0c-dbbcaa8a275a-config-file\") pod \"limitador-limitador-7c9b87d574-lzfhp\" (UID: \"77d9131c-32d0-4962-9e0c-dbbcaa8a275a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:19.201248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.201212    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2bq\" (UniqueName: \"kubernetes.io/projected/77d9131c-32d0-4962-9e0c-dbbcaa8a275a-kube-api-access-kr2bq\") pod \"limitador-limitador-7c9b87d574-lzfhp\" (UID: \"77d9131c-32d0-4962-9e0c-dbbcaa8a275a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:19.201444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.201269    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77d9131c-32d0-4962-9e0c-dbbcaa8a275a-config-file\") pod \"limitador-limitador-7c9b87d574-lzfhp\" (UID: \"77d9131c-32d0-4962-9e0c-dbbcaa8a275a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:19.201892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.201873    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77d9131c-32d0-4962-9e0c-dbbcaa8a275a-config-file\") pod \"limitador-limitador-7c9b87d574-lzfhp\" (UID: \"77d9131c-32d0-4962-9e0c-dbbcaa8a275a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:19.213918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.213891    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2bq\" (UniqueName: \"kubernetes.io/projected/77d9131c-32d0-4962-9e0c-dbbcaa8a275a-kube-api-access-kr2bq\") pod \"limitador-limitador-7c9b87d574-lzfhp\" (UID: \"77d9131c-32d0-4962-9e0c-dbbcaa8a275a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:19.344690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.344659    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:19.605180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.605105    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nlw6w"]
Oct 30 19:15:19.609084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.609047    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w"
Oct 30 19:15:19.614100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.614076    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5tbl8\""
Oct 30 19:15:19.615743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.615715    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nlw6w"]
Oct 30 19:15:19.619928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.619902    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:15:19.626085 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:15:19.624708    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d9131c_32d0_4962_9e0c_dbbcaa8a275a.slice/crio-66bf891a8f25854fe33af809552401583b05a74cf9aeda4e069901d45340c3e3 WatchSource:0}: Error finding container 66bf891a8f25854fe33af809552401583b05a74cf9aeda4e069901d45340c3e3: Status 404 returned error can't find the container with id 66bf891a8f25854fe33af809552401583b05a74cf9aeda4e069901d45340c3e3
Oct 30 19:15:19.706786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.706746    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr58r\" (UniqueName: \"kubernetes.io/projected/ab66237a-5f1f-46c9-933d-0b0575fbc17b-kube-api-access-hr58r\") pod \"authorino-56c6c9ddd5-nlw6w\" (UID: \"ab66237a-5f1f-46c9-933d-0b0575fbc17b\") " pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w"
Oct 30 19:15:19.808233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.808203    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr58r\" (UniqueName: \"kubernetes.io/projected/ab66237a-5f1f-46c9-933d-0b0575fbc17b-kube-api-access-hr58r\") pod \"authorino-56c6c9ddd5-nlw6w\" (UID: \"ab66237a-5f1f-46c9-933d-0b0575fbc17b\") " pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w"
Oct 30 19:15:19.821596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.821562    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr58r\" (UniqueName: \"kubernetes.io/projected/ab66237a-5f1f-46c9-933d-0b0575fbc17b-kube-api-access-hr58r\") pod \"authorino-56c6c9ddd5-nlw6w\" (UID: \"ab66237a-5f1f-46c9-933d-0b0575fbc17b\") " pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w"
Oct 30 19:15:19.923702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:19.923623    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w"
Oct 30 19:15:20.193752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:20.193725    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nlw6w"]
Oct 30 19:15:20.256315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:20.256278    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp" event={"ID":"77d9131c-32d0-4962-9e0c-dbbcaa8a275a","Type":"ContainerStarted","Data":"66bf891a8f25854fe33af809552401583b05a74cf9aeda4e069901d45340c3e3"}
Oct 30 19:15:20.257367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:20.257345    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w" event={"ID":"ab66237a-5f1f-46c9-933d-0b0575fbc17b","Type":"ContainerStarted","Data":"06a306ee67645721f66ba26f337a72a52081d5dde7266993d37efcfc9623424d"}
Oct 30 19:15:22.268202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:22.268162    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp" event={"ID":"77d9131c-32d0-4962-9e0c-dbbcaa8a275a","Type":"ContainerStarted","Data":"5451ce7dbef16b5866ef43b6f218b5c37078342a49392bdda194ec761a0c5093"}
Oct 30 19:15:22.268672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:22.268379    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:22.288257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:22.288204    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp" podStartSLOduration=1.639958378 podStartE2EDuration="3.288186271s" totalImagesPullingTime="1.648227893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:22.287142128 +0000 UTC m=+504.284057515" watchObservedRunningTime="2026-10-30 19:15:22.288186271 +0000 UTC m=+504.285101658"
Oct 30 19:15:23.274385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:23.274348    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w" event={"ID":"ab66237a-5f1f-46c9-933d-0b0575fbc17b","Type":"ContainerStarted","Data":"fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0"}
Oct 30 19:15:23.291861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:23.291809    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w" podStartSLOduration=1.902322326 podStartE2EDuration="4.291796011s" totalImagesPullingTime="2.389473685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:23.290423362 +0000 UTC m=+505.287338745" watchObservedRunningTime="2026-10-30 19:15:23.291796011 +0000 UTC m=+505.288711386"
Oct 30 19:15:33.275926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:33.275889    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"
Oct 30 19:15:43.638913 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:15:43.638869    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:15:53.739859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:53.739821    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nlw6w"]
Oct 30 19:15:53.740428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:53.740022    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w" podUID="ab66237a-5f1f-46c9-933d-0b0575fbc17b" containerName="authorino" containerID="cri-o://fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0" gracePeriod=30
Oct 30 19:15:53.987967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:53.987941    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w"
Oct 30 19:15:54.021370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.021286    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab66237a-5f1f-46c9-933d-0b0575fbc17b-kube-api-access-hr58r\" (UniqueName: \"kubernetes.io/projected/ab66237a-5f1f-46c9-933d-0b0575fbc17b-kube-api-access-hr58r\") pod \"ab66237a-5f1f-46c9-933d-0b0575fbc17b\" (UID: \"ab66237a-5f1f-46c9-933d-0b0575fbc17b\") "
Oct 30 19:15:54.023311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.023285    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab66237a-5f1f-46c9-933d-0b0575fbc17b-kube-api-access-hr58r" pod "ab66237a-5f1f-46c9-933d-0b0575fbc17b" (UID: "ab66237a-5f1f-46c9-933d-0b0575fbc17b"). InnerVolumeSpecName "kube-api-access-hr58r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:54.121950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.121922    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr58r\" (UniqueName: \"kubernetes.io/projected/ab66237a-5f1f-46c9-933d-0b0575fbc17b-kube-api-access-hr58r\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:54.408583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.408542    6777 generic.go:356] "Generic (PLEG): container finished" podID="ab66237a-5f1f-46c9-933d-0b0575fbc17b" containerID="fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0" exitCode=0
Oct 30 19:15:54.408583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.408583    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w" event={"ID":"ab66237a-5f1f-46c9-933d-0b0575fbc17b","Type":"ContainerDied","Data":"fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0"}
Oct 30 19:15:54.408855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.408595    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w"
Oct 30 19:15:54.408855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.408612    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nlw6w" event={"ID":"ab66237a-5f1f-46c9-933d-0b0575fbc17b","Type":"ContainerDied","Data":"06a306ee67645721f66ba26f337a72a52081d5dde7266993d37efcfc9623424d"}
Oct 30 19:15:54.408855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.408629    6777 scope.go:122] "RemoveContainer" containerID="fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0"
Oct 30 19:15:54.417027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.417010    6777 scope.go:122] "RemoveContainer" containerID="fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0"
Oct 30 19:15:54.417310 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:15:54.417293    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0\": container with ID starting with fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0 not found: ID does not exist" containerID="fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0"
Oct 30 19:15:54.417359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.417317    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0"} err="failed to get container status \"fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0\": rpc error: code = NotFound desc = could not find container \"fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0\": container with ID starting with fca5d1b58f83e65d31428a0b91904f159f7bebad1fb0a9a297f60cdecc0fc9c0 not found: ID does not exist"
Oct 30 19:15:54.425822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.425799    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nlw6w"]
Oct 30 19:15:54.432152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:54.432129    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nlw6w"]
Oct 30 19:15:56.112327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:15:56.112292    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab66237a-5f1f-46c9-933d-0b0575fbc17b" path="/var/lib/kubelet/pods/ab66237a-5f1f-46c9-933d-0b0575fbc17b/volumes"
Oct 30 19:16:13.639104 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:16:13.639014    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:16:37.119163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:16:37.119124    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:16:43.638709 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:16:43.638668    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:17:13.638304 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:17:13.638267    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:17:43.431186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:43.431151    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:17:43.638612 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:17:43.638578    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:17:45.686859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.686820    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:17:45.766286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.766257    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"]
Oct 30 19:17:45.766660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.766646    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ab66237a-5f1f-46c9-933d-0b0575fbc17b" containerName="authorino"
Oct 30 19:17:45.766713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.766662    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab66237a-5f1f-46c9-933d-0b0575fbc17b" containerName="authorino"
Oct 30 19:17:45.766753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.766737    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab66237a-5f1f-46c9-933d-0b0575fbc17b" containerName="authorino"
Oct 30 19:17:45.770011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.769990    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:45.774864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.774845    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 19:17:45.776111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.776087    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 19:17:45.776248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.776123    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fvzqm\""
Oct 30 19:17:45.776248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.776172    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 19:17:45.779117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.779092    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"]
Oct 30 19:17:45.915512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.915480    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:45.915512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.915512    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:45.915720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.915539    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:45.915720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.915679    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:45.915794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.915720    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:45.915794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:45.915751    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmwf\" (UniqueName: \"kubernetes.io/projected/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-kube-api-access-rtmwf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.016856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.016777    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.016856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.016817    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.016856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.016840    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.017123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.016887    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.017123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.016913    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.017123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.016946    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmwf\" (UniqueName: \"kubernetes.io/projected/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-kube-api-access-rtmwf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.017232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.017200    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.017276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.017255    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.017374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.017349    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.019206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.019185    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.019498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.019478    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.029522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.029499    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmwf\" (UniqueName: \"kubernetes.io/projected/d3d7cb45-4e6d-466b-aa7b-498e2a682c99-kube-api-access-rtmwf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf\" (UID: \"d3d7cb45-4e6d-466b-aa7b-498e2a682c99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.083870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.083831    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:46.364113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.364082    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"]
Oct 30 19:17:46.367097 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:17:46.367050    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d7cb45_4e6d_466b_aa7b_498e2a682c99.slice/crio-d7ed550ceec4de8cbaa1e553cef7e70b1fac5c0e0c4112f03f3df7106aa8d66e WatchSource:0}: Error finding container d7ed550ceec4de8cbaa1e553cef7e70b1fac5c0e0c4112f03f3df7106aa8d66e: Status 404 returned error can't find the container with id d7ed550ceec4de8cbaa1e553cef7e70b1fac5c0e0c4112f03f3df7106aa8d66e
Oct 30 19:17:46.368411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.368395    6777 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:17:46.641493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.641408    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"]
Oct 30 19:17:46.646173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.646150    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.655632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.654768    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"]
Oct 30 19:17:46.656169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.656134    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 19:17:46.724145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.723714    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrgd\" (UniqueName: \"kubernetes.io/projected/d9cbeb26-fde8-410b-b600-b3d536fed7c8-kube-api-access-pfrgd\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.724145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.723781    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.724145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.723841    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.724145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.723868    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.724145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.723929    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9cbeb26-fde8-410b-b600-b3d536fed7c8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.724145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.723966    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.824724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.824682    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.824900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.824737    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.824900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.824791    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9cbeb26-fde8-410b-b600-b3d536fed7c8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.824900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.824817    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.824900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.824861    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrgd\" (UniqueName: \"kubernetes.io/projected/d9cbeb26-fde8-410b-b600-b3d536fed7c8-kube-api-access-pfrgd\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.824900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.824894    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.825363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.825340    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.825571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.825548    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.825748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.825730    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.828704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.828678    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9cbeb26-fde8-410b-b600-b3d536fed7c8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.829323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.829296    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9cbeb26-fde8-410b-b600-b3d536fed7c8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.839665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.839638    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrgd\" (UniqueName: \"kubernetes.io/projected/d9cbeb26-fde8-410b-b600-b3d536fed7c8-kube-api-access-pfrgd\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28\" (UID: \"d9cbeb26-fde8-410b-b600-b3d536fed7c8\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:46.902434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.902344    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf" event={"ID":"d3d7cb45-4e6d-466b-aa7b-498e2a682c99","Type":"ContainerStarted","Data":"d7ed550ceec4de8cbaa1e553cef7e70b1fac5c0e0c4112f03f3df7106aa8d66e"}
Oct 30 19:17:46.961704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:46.961663    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:47.253045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:47.252945    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:17:47.256807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:47.256783    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"]
Oct 30 19:17:47.259800 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:17:47.259765    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9cbeb26_fde8_410b_b600_b3d536fed7c8.slice/crio-8a9a10b6bd2801274af31c8b5bec2a3c854830546ba91e5227585f2771a63b47 WatchSource:0}: Error finding container 8a9a10b6bd2801274af31c8b5bec2a3c854830546ba91e5227585f2771a63b47: Status 404 returned error can't find the container with id 8a9a10b6bd2801274af31c8b5bec2a3c854830546ba91e5227585f2771a63b47
Oct 30 19:17:47.908113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:47.908012    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf" event={"ID":"d3d7cb45-4e6d-466b-aa7b-498e2a682c99","Type":"ContainerStarted","Data":"b01afd555e7a47802f16a4ff5a1d6cbbccb07cb7d03891d2dc53a4a12cb0d7c8"}
Oct 30 19:17:47.909680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:47.909656    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" event={"ID":"d9cbeb26-fde8-410b-b600-b3d536fed7c8","Type":"ContainerStarted","Data":"40521cda47a5490b2fb67a183005250a1f5715bfaaae2feef835c5a94be295cf"}
Oct 30 19:17:47.909796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:47.909693    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" event={"ID":"d9cbeb26-fde8-410b-b600-b3d536fed7c8","Type":"ContainerStarted","Data":"8a9a10b6bd2801274af31c8b5bec2a3c854830546ba91e5227585f2771a63b47"}
Oct 30 19:17:47.933292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:47.933245    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf" podStartSLOduration=1.7316026519999999 podStartE2EDuration="2.933231441s" totalImagesPullingTime="1.201628789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:47.931723982 +0000 UTC m=+649.928639364" watchObservedRunningTime="2026-10-30 19:17:47.933231441 +0000 UTC m=+649.930146825"
Oct 30 19:17:47.957711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:47.957642    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podStartSLOduration=1.6491478769999999 podStartE2EDuration="1.957625718s" totalImagesPullingTime="308.477841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:47.957185966 +0000 UTC m=+649.954101364" watchObservedRunningTime="2026-10-30 19:17:47.957625718 +0000 UTC m=+649.954541103"
Oct 30 19:17:47.962387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:47.962364    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:48.084504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:48.084462    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:49.060809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.060774    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"]
Oct 30 19:17:49.065031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.065009    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.071464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.071442    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 19:17:49.075308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.075185    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"]
Oct 30 19:17:49.153357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.153322    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c45319f-1d63-435a-85b6-5e49094db5be-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.153525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.153375    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.153525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.153435    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.153600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.153529    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qghdj\" (UniqueName: \"kubernetes.io/projected/0c45319f-1d63-435a-85b6-5e49094db5be-kube-api-access-qghdj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.153600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.153583    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.153666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.153625    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.254351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.254313    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qghdj\" (UniqueName: \"kubernetes.io/projected/0c45319f-1d63-435a-85b6-5e49094db5be-kube-api-access-qghdj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.254506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.254484    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.254594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.254519    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.254594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.254564    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c45319f-1d63-435a-85b6-5e49094db5be-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.254714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.254630    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.254714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.254656    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.254984 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.254949    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.255224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.255201    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.255322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.255277    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.257018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.256987    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c45319f-1d63-435a-85b6-5e49094db5be-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.257203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.257186    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c45319f-1d63-435a-85b6-5e49094db5be-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.272229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.272207    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qghdj\" (UniqueName: \"kubernetes.io/projected/0c45319f-1d63-435a-85b6-5e49094db5be-kube-api-access-qghdj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mlz74\" (UID: \"0c45319f-1d63-435a-85b6-5e49094db5be\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.381150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.381036    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:49.657725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.657244    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"]
Oct 30 19:17:49.660026 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:17:49.659989    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c45319f_1d63_435a_85b6_5e49094db5be.slice/crio-c03affb87afd2ea52a28970992240f2d92c5a6ccbf1147e6904abbb4dbfada47 WatchSource:0}: Error finding container c03affb87afd2ea52a28970992240f2d92c5a6ccbf1147e6904abbb4dbfada47: Status 404 returned error can't find the container with id c03affb87afd2ea52a28970992240f2d92c5a6ccbf1147e6904abbb4dbfada47
Oct 30 19:17:49.919859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:49.919771    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74" event={"ID":"0c45319f-1d63-435a-85b6-5e49094db5be","Type":"ContainerStarted","Data":"c03affb87afd2ea52a28970992240f2d92c5a6ccbf1147e6904abbb4dbfada47"}
Oct 30 19:17:50.064000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:50.063966    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:17:50.925857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:50.925814    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74" event={"ID":"0c45319f-1d63-435a-85b6-5e49094db5be","Type":"ContainerStarted","Data":"4b90eb9d4cbc9fbc19125ff729231e518023c8e557e9b1a2647590f39e169edb"}
Oct 30 19:17:50.947723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:50.947673    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74" podStartSLOduration=1.658371704 podStartE2EDuration="1.947659826s" totalImagesPullingTime="289.288122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:50.946501302 +0000 UTC m=+652.943416689" watchObservedRunningTime="2026-10-30 19:17:50.947659826 +0000 UTC m=+652.944575207"
Oct 30 19:17:51.381270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:51.381221    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:53.349222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:53.349188    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:17:56.084564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:56.084528    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:56.097049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:56.097022    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:56.962536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:56.962503    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:56.963951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:56.963923    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf"
Oct 30 19:17:56.975634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:56.975610    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:57.968364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:57.968334    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:17:59.381896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:59.381857    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:59.394491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:59.394467    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:17:59.980443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:17:59.980405    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74"
Oct 30 19:18:03.165836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.165801    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-757f58996d-sl6pb"]
Oct 30 19:18:03.170847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.170825    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757f58996d-sl6pb"
Oct 30 19:18:03.171536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.171512    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/043e9e20-e8e8-4403-a5d9-85cfd385d5cd-tls-cert\") pod \"authorino-757f58996d-sl6pb\" (UID: \"043e9e20-e8e8-4403-a5d9-85cfd385d5cd\") " pod="kuadrant-system/authorino-757f58996d-sl6pb"
Oct 30 19:18:03.171616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.171545    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwkb\" (UniqueName: \"kubernetes.io/projected/043e9e20-e8e8-4403-a5d9-85cfd385d5cd-kube-api-access-tqwkb\") pod \"authorino-757f58996d-sl6pb\" (UID: \"043e9e20-e8e8-4403-a5d9-85cfd385d5cd\") " pod="kuadrant-system/authorino-757f58996d-sl6pb"
Oct 30 19:18:03.176813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.176783    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-757f58996d-sl6pb"]
Oct 30 19:18:03.177008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.176990    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 19:18:03.177118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.177103    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5tbl8\""
Oct 30 19:18:03.272025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.271989    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/043e9e20-e8e8-4403-a5d9-85cfd385d5cd-tls-cert\") pod \"authorino-757f58996d-sl6pb\" (UID: \"043e9e20-e8e8-4403-a5d9-85cfd385d5cd\") " pod="kuadrant-system/authorino-757f58996d-sl6pb"
Oct 30 19:18:03.272025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.272026    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwkb\" (UniqueName: \"kubernetes.io/projected/043e9e20-e8e8-4403-a5d9-85cfd385d5cd-kube-api-access-tqwkb\") pod \"authorino-757f58996d-sl6pb\" (UID: \"043e9e20-e8e8-4403-a5d9-85cfd385d5cd\") " pod="kuadrant-system/authorino-757f58996d-sl6pb"
Oct 30 19:18:03.274436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.274410    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/043e9e20-e8e8-4403-a5d9-85cfd385d5cd-tls-cert\") pod \"authorino-757f58996d-sl6pb\" (UID: \"043e9e20-e8e8-4403-a5d9-85cfd385d5cd\") " pod="kuadrant-system/authorino-757f58996d-sl6pb"
Oct 30 19:18:03.284746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.284716    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwkb\" (UniqueName: \"kubernetes.io/projected/043e9e20-e8e8-4403-a5d9-85cfd385d5cd-kube-api-access-tqwkb\") pod \"authorino-757f58996d-sl6pb\" (UID: \"043e9e20-e8e8-4403-a5d9-85cfd385d5cd\") " pod="kuadrant-system/authorino-757f58996d-sl6pb"
Oct 30 19:18:03.484054 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.483963    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757f58996d-sl6pb"
Oct 30 19:18:03.756253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.756091    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-757f58996d-sl6pb"]
Oct 30 19:18:03.759033 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:18:03.758986    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod043e9e20_e8e8_4403_a5d9_85cfd385d5cd.slice/crio-3d52b4202f0d02fb3c54528f03bd23bd041d81b359ea558cb03512f4611e85a9 WatchSource:0}: Error finding container 3d52b4202f0d02fb3c54528f03bd23bd041d81b359ea558cb03512f4611e85a9: Status 404 returned error can't find the container with id 3d52b4202f0d02fb3c54528f03bd23bd041d81b359ea558cb03512f4611e85a9
Oct 30 19:18:03.986435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:03.986400    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757f58996d-sl6pb" event={"ID":"043e9e20-e8e8-4403-a5d9-85cfd385d5cd","Type":"ContainerStarted","Data":"3d52b4202f0d02fb3c54528f03bd23bd041d81b359ea558cb03512f4611e85a9"}
Oct 30 19:18:04.992343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:04.992309    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757f58996d-sl6pb" event={"ID":"043e9e20-e8e8-4403-a5d9-85cfd385d5cd","Type":"ContainerStarted","Data":"ee459feb0b9b051523aac585219dc18a0cfec126e023714b27fa82fc20876038"}
Oct 30 19:18:05.011110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:05.010962    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-757f58996d-sl6pb" podStartSLOduration=1.025658585 podStartE2EDuration="2.010948865s" totalImagesPullingTime="985.29028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:05.010517866 +0000 UTC m=+667.007433252" watchObservedRunningTime="2026-10-30 19:18:05.010948865 +0000 UTC m=+667.007864249"
Oct 30 19:18:07.446558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.446524    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"]
Oct 30 19:18:07.453011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.451230    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.461751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.461708    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:18:07.463760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.463732    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"]
Oct 30 19:18:07.517705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.517666    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.518003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.517971    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.518233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.518017    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.518233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.518041    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nqqm\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-kube-api-access-8nqqm\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.518233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.518083    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.518233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.518105    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.518233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.518129    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.518233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.518149    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de2b7b1-50d6-400c-973c-f9e5c076042b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.518233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.518198    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.541076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.540242    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vdgcw\""
Oct 30 19:18:07.618803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.618750    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.618803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.618801    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.618822    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.618842    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nqqm\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-kube-api-access-8nqqm\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.618867    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.618884    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.618900    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.618923    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de2b7b1-50d6-400c-973c-f9e5c076042b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.618956    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.619409    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.619477    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.619410    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.619821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.619793    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.620134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.620111    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de2b7b1-50d6-400c-973c-f9e5c076042b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.626092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.621894    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.626092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.622491    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.640351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.640318    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.641956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.641919    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nqqm\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-kube-api-access-8nqqm\") pod \"maas-default-gateway-openshift-default-855f9574cf-fpplz\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:07.776155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:07.776070    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:08.055291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:08.055254    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"]
Oct 30 19:18:08.058559 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:18:08.058529    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de2b7b1_50d6_400c_973c_f9e5c076042b.slice/crio-00bfcc65836af40ce11ed55351631f5ed5a38a0ac70ce153b1582f37e3e76c7f WatchSource:0}: Error finding container 00bfcc65836af40ce11ed55351631f5ed5a38a0ac70ce153b1582f37e3e76c7f: Status 404 returned error can't find the container with id 00bfcc65836af40ce11ed55351631f5ed5a38a0ac70ce153b1582f37e3e76c7f
Oct 30 19:18:09.021710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:09.021671    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" event={"ID":"0de2b7b1-50d6-400c-973c-f9e5c076042b","Type":"ContainerStarted","Data":"00bfcc65836af40ce11ed55351631f5ed5a38a0ac70ce153b1582f37e3e76c7f"}
Oct 30 19:18:11.310396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.310318    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:18:11.310396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.310388    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:18:11.310718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.310421    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:18:11.474820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.474784    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-gblqx"]
Oct 30 19:18:11.483443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.483407    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:11.491943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.491850    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-gblqx"]
Oct 30 19:18:11.493122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.493098    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 19:18:11.493122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.493098    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vsg9g\""
Oct 30 19:18:11.497358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.497314    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"]
Oct 30 19:18:11.501319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.501296    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:11.515150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.515110    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"]
Oct 30 19:18:11.557180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.557144    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d7f713-e99d-47f0-a335-759341ccd335-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-svc7x\" (UID: \"d5d7f713-e99d-47f0-a335-759341ccd335\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:11.557329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.557192    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9mh\" (UniqueName: \"kubernetes.io/projected/2bf535e3-96e0-42c7-beef-f525cf094dae-kube-api-access-4h9mh\") pod \"payload-processing-5f78687d5b-gblqx\" (UID: \"2bf535e3-96e0-42c7-beef-f525cf094dae\") " pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:11.557388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.557373    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8gl\" (UniqueName: \"kubernetes.io/projected/d5d7f713-e99d-47f0-a335-759341ccd335-kube-api-access-zm8gl\") pod \"payload-pre-processing-7b98bb678c-svc7x\" (UID: \"d5d7f713-e99d-47f0-a335-759341ccd335\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:11.557424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.557407    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf535e3-96e0-42c7-beef-f525cf094dae-plugins-config-volume\") pod \"payload-processing-5f78687d5b-gblqx\" (UID: \"2bf535e3-96e0-42c7-beef-f525cf094dae\") " pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:11.658212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.658110    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8gl\" (UniqueName: \"kubernetes.io/projected/d5d7f713-e99d-47f0-a335-759341ccd335-kube-api-access-zm8gl\") pod \"payload-pre-processing-7b98bb678c-svc7x\" (UID: \"d5d7f713-e99d-47f0-a335-759341ccd335\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:11.658212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.658176    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf535e3-96e0-42c7-beef-f525cf094dae-plugins-config-volume\") pod \"payload-processing-5f78687d5b-gblqx\" (UID: \"2bf535e3-96e0-42c7-beef-f525cf094dae\") " pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:11.658432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.658223    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d7f713-e99d-47f0-a335-759341ccd335-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-svc7x\" (UID: \"d5d7f713-e99d-47f0-a335-759341ccd335\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:11.658432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.658264    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9mh\" (UniqueName: \"kubernetes.io/projected/2bf535e3-96e0-42c7-beef-f525cf094dae-kube-api-access-4h9mh\") pod \"payload-processing-5f78687d5b-gblqx\" (UID: \"2bf535e3-96e0-42c7-beef-f525cf094dae\") " pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:11.658947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.658922    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d7f713-e99d-47f0-a335-759341ccd335-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-svc7x\" (UID: \"d5d7f713-e99d-47f0-a335-759341ccd335\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:11.659172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.659154    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf535e3-96e0-42c7-beef-f525cf094dae-plugins-config-volume\") pod \"payload-processing-5f78687d5b-gblqx\" (UID: \"2bf535e3-96e0-42c7-beef-f525cf094dae\") " pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:11.670996 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.670952    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8gl\" (UniqueName: \"kubernetes.io/projected/d5d7f713-e99d-47f0-a335-759341ccd335-kube-api-access-zm8gl\") pod \"payload-pre-processing-7b98bb678c-svc7x\" (UID: \"d5d7f713-e99d-47f0-a335-759341ccd335\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:11.671136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.671040    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9mh\" (UniqueName: \"kubernetes.io/projected/2bf535e3-96e0-42c7-beef-f525cf094dae-kube-api-access-4h9mh\") pod \"payload-processing-5f78687d5b-gblqx\" (UID: \"2bf535e3-96e0-42c7-beef-f525cf094dae\") " pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:11.801003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.800972    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:11.819010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:11.818974    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:12.038138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:12.038099    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" event={"ID":"0de2b7b1-50d6-400c-973c-f9e5c076042b","Type":"ContainerStarted","Data":"6bc5662f5d2b5e96112073af4136d372f88ecd9fa388f33f5228d8bbc2426589"}
Oct 30 19:18:12.098667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:12.098622    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" podStartSLOduration=1.8498613929999999 podStartE2EDuration="5.098606125s" totalImagesPullingTime="3.248744732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:12.073578921 +0000 UTC m=+674.070494307" watchObservedRunningTime="2026-10-30 19:18:12.098606125 +0000 UTC m=+674.095521510"
Oct 30 19:18:12.099006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:12.098987    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-gblqx"]
Oct 30 19:18:12.100942 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:18:12.100916    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf535e3_96e0_42c7_beef_f525cf094dae.slice/crio-803e781b1c3809808d92951c0a5563570a60d4daec415edad690450129f79ac6 WatchSource:0}: Error finding container 803e781b1c3809808d92951c0a5563570a60d4daec415edad690450129f79ac6: Status 404 returned error can't find the container with id 803e781b1c3809808d92951c0a5563570a60d4daec415edad690450129f79ac6
Oct 30 19:18:12.125540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:12.125517    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"]
Oct 30 19:18:12.128103 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:18:12.128076    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d7f713_e99d_47f0_a335_759341ccd335.slice/crio-f233b7c305dacd7da8c36448daddf8407a36bb9453214da964ae79f2fe5f7442 WatchSource:0}: Error finding container f233b7c305dacd7da8c36448daddf8407a36bb9453214da964ae79f2fe5f7442: Status 404 returned error can't find the container with id f233b7c305dacd7da8c36448daddf8407a36bb9453214da964ae79f2fe5f7442
Oct 30 19:18:12.776255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:12.776219    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:12.779219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:12.778749    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fpplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:18:12.779219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:12.778844    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:18:13.045761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:13.045641    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-gblqx" event={"ID":"2bf535e3-96e0-42c7-beef-f525cf094dae","Type":"ContainerStarted","Data":"803e781b1c3809808d92951c0a5563570a60d4daec415edad690450129f79ac6"}
Oct 30 19:18:13.047725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:13.047695    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x" event={"ID":"d5d7f713-e99d-47f0-a335-759341ccd335","Type":"ContainerStarted","Data":"f233b7c305dacd7da8c36448daddf8407a36bb9453214da964ae79f2fe5f7442"}
Oct 30 19:18:13.647302 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:18:13.647264    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:18:13.777074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:13.776973    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fpplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:18:13.777074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:13.777027    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:18:14.777200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:14.776704    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fpplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:18:14.777200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:14.776772    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:18:15.777351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:15.777257    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fpplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:18:15.777351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:15.777327    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:18:16.099806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:16.099753    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x" event={"ID":"d5d7f713-e99d-47f0-a335-759341ccd335","Type":"ContainerStarted","Data":"39f4acb8b994a847463ce7e3081cf44028da7943f5160314ce69753191952cc6"}
Oct 30 19:18:16.100093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:16.099908    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:16.101925 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:16.101896    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-gblqx" event={"ID":"2bf535e3-96e0-42c7-beef-f525cf094dae","Type":"ContainerStarted","Data":"23c4bbe4889f64acf2c2210ca0487cc5a1f6035e2b545c34d63b6debc73f9252"}
Oct 30 19:18:16.102052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:16.102036    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:16.120774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:16.120727    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x" podStartSLOduration=1.9500692929999999 podStartE2EDuration="5.120713522s" totalImagesPullingTime="3.170644229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:16.11859804 +0000 UTC m=+678.115513426" watchObservedRunningTime="2026-10-30 19:18:16.120713522 +0000 UTC m=+678.117628905"
Oct 30 19:18:16.174076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:16.174001    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-gblqx" podStartSLOduration=1.865889809 podStartE2EDuration="5.173985082s" totalImagesPullingTime="3.308095273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:16.173475047 +0000 UTC m=+678.170390444" watchObservedRunningTime="2026-10-30 19:18:16.173985082 +0000 UTC m=+678.170900465"
Oct 30 19:18:16.776985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:16.776942    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fpplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:18:16.777278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:16.777003    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:18:17.776929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:17.776891    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fpplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:18:17.777339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:17.776958    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:18:18.777114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:18.777052    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fpplz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:18:18.777509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:18.777134    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:18:19.781459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:19.781431    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:19.781843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:19.781728    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:19.782409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:19.782392    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:18:22.468137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.468103    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"]
Oct 30 19:18:22.474579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.474557    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.485091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.485048    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"]
Oct 30 19:18:22.488162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.488137    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"]
Oct 30 19:18:22.493051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.493031    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.503800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.503772    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"]
Oct 30 19:18:22.586231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586196    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.586231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586228    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.586438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586250    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdjln\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-kube-api-access-tdjln\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.586438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586359    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.586438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586380    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.586438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586402    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.586438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586431    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.586594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586534    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.586594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586570    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cc7h\" (UniqueName: \"kubernetes.io/projected/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-kube-api-access-9cc7h\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.586652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586621    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.586684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586656    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.586684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586679    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.586743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586701    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.586743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586735    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.586801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586787    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014ae06d-ddaf-41b9-a59a-543047e656fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.586846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586833    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.586882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586858    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.586914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.586881    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688012    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cc7h\" (UniqueName: \"kubernetes.io/projected/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-kube-api-access-9cc7h\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688084    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688132    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688152    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688175    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688216    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688243    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014ae06d-ddaf-41b9-a59a-543047e656fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688273    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688302    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688333    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688398    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688423    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688448    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdjln\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-kube-api-access-tdjln\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688484    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688516    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688546    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688590    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688597    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688606    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688642    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.688929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688648    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688813    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.688929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688848    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.689290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.688947    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.689290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.689008    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014ae06d-ddaf-41b9-a59a-543047e656fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.689290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.689133    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.689290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.689204    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.689537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.689518    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.690840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.690815    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.691346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.691324    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.691463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.691330    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.691463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.691384    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.701716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.701690    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.702119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.702097    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.702272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.702252    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdjln\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-kube-api-access-tdjln\") pod \"maas-default-gateway-openshift-default-855f9574cf-chjvh\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.702322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.702305    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cc7h\" (UniqueName: \"kubernetes.io/projected/8084f1f2-0fb7-4dbb-b255-4ffcef554e9c-kube-api-access-9cc7h\") pod \"maas-default-gateway-openshift-default-855f9574cf-pthrg\" (UID: \"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:22.791191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.791157    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:22.811227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:22.811200    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:23.080888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.080863    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"]
Oct 30 19:18:23.083095 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:18:23.083043    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014ae06d_ddaf_41b9_a59a_543047e656fd.slice/crio-abc4b047e3624b7772f06d6ea07124ae414a99ceb17f07eef701bf317b61726c WatchSource:0}: Error finding container abc4b047e3624b7772f06d6ea07124ae414a99ceb17f07eef701bf317b61726c: Status 404 returned error can't find the container with id abc4b047e3624b7772f06d6ea07124ae414a99ceb17f07eef701bf317b61726c
Oct 30 19:18:23.086299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.086268    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:18:23.086392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.086328    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:18:23.086392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.086357    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:18:23.102908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.102883    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"]
Oct 30 19:18:23.118214 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:18:23.118190    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8084f1f2_0fb7_4dbb_b255_4ffcef554e9c.slice/crio-4a8f708b0434566c793bae616cf69f3ad0688c59156f77d9e0932e3a3caf4a9a WatchSource:0}: Error finding container 4a8f708b0434566c793bae616cf69f3ad0688c59156f77d9e0932e3a3caf4a9a: Status 404 returned error can't find the container with id 4a8f708b0434566c793bae616cf69f3ad0688c59156f77d9e0932e3a3caf4a9a
Oct 30 19:18:23.121364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.121334    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:18:23.121457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.121397    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:18:23.121457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.121426    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:18:23.137261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.137234    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" event={"ID":"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c","Type":"ContainerStarted","Data":"4a8f708b0434566c793bae616cf69f3ad0688c59156f77d9e0932e3a3caf4a9a"}
Oct 30 19:18:23.138545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:23.138523    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" event={"ID":"014ae06d-ddaf-41b9-a59a-543047e656fd","Type":"ContainerStarted","Data":"abc4b047e3624b7772f06d6ea07124ae414a99ceb17f07eef701bf317b61726c"}
Oct 30 19:18:24.144212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.144160    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" event={"ID":"8084f1f2-0fb7-4dbb-b255-4ffcef554e9c","Type":"ContainerStarted","Data":"30af0159e9db10dc1bc2e42e78b3ef865da49ee3a98ea3174f33b80bfc3615b7"}
Oct 30 19:18:24.145835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.145805    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" event={"ID":"014ae06d-ddaf-41b9-a59a-543047e656fd","Type":"ContainerStarted","Data":"c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572"}
Oct 30 19:18:24.213724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.213668    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" podStartSLOduration=2.213650747 podStartE2EDuration="2.213650747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:24.212885728 +0000 UTC m=+686.209801114" watchObservedRunningTime="2026-10-30 19:18:24.213650747 +0000 UTC m=+686.210566188"
Oct 30 19:18:24.213902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.213870    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podStartSLOduration=2.213865756 podStartE2EDuration="2.213865756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:24.177905535 +0000 UTC m=+686.174820919" watchObservedRunningTime="2026-10-30 19:18:24.213865756 +0000 UTC m=+686.210781136"
Oct 30 19:18:24.795152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.795113    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:24.797188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.797161    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-chjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:18:24.797383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.797364    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:18:24.811686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.811665    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:24.812601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.812578    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:18:24.812691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:24.812623    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 19:18:25.794081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:25.793528    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-chjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:18:25.794081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:25.793597    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:18:25.811806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:25.811772    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:18:25.812009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:25.811829    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 19:18:26.793577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:26.793536    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-chjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:18:26.793880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:26.793851    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:18:26.812226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:26.812183    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:18:26.812589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:26.812239    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 19:18:27.109592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:27.109515    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-svc7x"
Oct 30 19:18:27.109888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:27.109867    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-gblqx"
Oct 30 19:18:27.797085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:27.795504    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-chjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:18:27.797085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:27.795576    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:18:27.811538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:27.811500    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:18:27.811707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:27.811547    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 19:18:28.798085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:28.797167    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-chjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:18:28.798085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:28.797247    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:18:28.811892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:28.811856    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:18:28.811980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:28.811941    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 19:18:29.791582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:29.791534    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-chjvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:18:29.791805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:29.791608    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:18:29.811907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:29.811862    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:18:29.812335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:29.811978    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 19:18:31.340644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:31.340607    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:31.341568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:31.341510    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:31.342401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:31.342381    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:18:31.380152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:31.380124    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:31.380798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:31.380772    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:31.381461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:18:31.381444    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg"
Oct 30 19:18:43.639126 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:18:43.639085    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:19:04.300886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.300850    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"]
Oct 30 19:19:04.312301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.312277    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.324118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.324089    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"]
Oct 30 19:19:04.324364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.324344    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cfa4bb-openshift-default-dockercfg-jqxm7\""
Oct 30 19:19:04.407399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.407366    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"]
Oct 30 19:19:04.411370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.411345    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.417021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.416995    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.417021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.417028    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.417193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.417049    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.417233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.417207    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.417233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.417224    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7327b5-03ab-49fa-89e7-08df93bf233d-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.417310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.417269    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.417399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.417378    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.417461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.417413    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c54\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-kube-api-access-r5c54\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.417534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.417514    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.428175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.428150    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-27b5d5-openshift-default-dockercfg-djg8x\""
Oct 30 19:19:04.428341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.428321    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"]
Oct 30 19:19:04.518189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518153    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-certs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.518378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518197    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.518378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518215    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7327b5-03ab-49fa-89e7-08df93bf233d-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.518378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518304    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.518378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518340    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-data\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.518547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518470    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hg84\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-kube-api-access-9hg84\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.518547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518507    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.518547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518534    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.518679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518557    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c54\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-kube-api-access-r5c54\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.518679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518610    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-podinfo\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.518679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518640    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518676    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518704    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518738    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518775    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a0608a-109b-4aef-88ab-52b53be4ca45-istiod-ca-cert\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518801    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-credential-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518858    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518846    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-envoy\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518883    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.518921    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7327b5-03ab-49fa-89e7-08df93bf233d-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.519180    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-token\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.519243    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.519923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.519268    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.521754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.521723    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.521921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.521902    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.552929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.552862    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c54\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-kube-api-access-r5c54\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.553070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.553041    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.620716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.620676    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a0608a-109b-4aef-88ab-52b53be4ca45-istiod-ca-cert\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.620907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.620723    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-credential-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.620907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.620866    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-envoy\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.620930    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-token\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.621015    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-certs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.621084    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.621173    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-credential-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.621225    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-data\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.621271    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hg84\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-kube-api-access-9hg84\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.621326    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.621380    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-podinfo\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621479 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.621420    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-certs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.621534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.621501    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a0608a-109b-4aef-88ab-52b53be4ca45-istiod-ca-cert\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.622197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.622169    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-data\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.624687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.624531    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-envoy\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.624890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.624865    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-podinfo\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.625257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.625054    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:04.643096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.643051    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-token\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.644012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.643989    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hg84\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-kube-api-access-9hg84\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.647481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.647460    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"]
Oct 30 19:19:04.652476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.652457    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.660471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.660447    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-17e904-openshift-default-dockercfg-d5z8k\""
Oct 30 19:19:04.667475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.667447    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"]
Oct 30 19:19:04.723176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.722583    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.723176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.722664    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.723176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.722690    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42325e6-a838-4786-9c57-08fa864e811a-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.723176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.722770    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.723176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.722804    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwr8s\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-kube-api-access-bwr8s\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.723176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.722834    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42325e6-a838-4786-9c57-08fa864e811a-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.723176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.722885    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.723176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.722909    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.723176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.722945    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.724610 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.724585    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:04.824828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.824722    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.824828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.824765    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42325e6-a838-4786-9c57-08fa864e811a-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.825105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.824853    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.825105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.824884    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwr8s\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-kube-api-access-bwr8s\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.825105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.824911    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42325e6-a838-4786-9c57-08fa864e811a-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.825105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.824963    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.825105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.824986    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.825105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.825024    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.825465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.825112    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.825571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.825547    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.825634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.825575    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.826111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.826084    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42325e6-a838-4786-9c57-08fa864e811a-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.826212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.826196    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.826353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.826328    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.828135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.828116    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.828664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.828642    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42325e6-a838-4786-9c57-08fa864e811a-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.844649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.844617    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.844758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.844615    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwr8s\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-kube-api-access-bwr8s\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:04.849871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.849847    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"]
Oct 30 19:19:04.854890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.854873    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.860693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.860672    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ad23c7-openshift-default-dockercfg-b7xt7\""
Oct 30 19:19:04.866692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.866666    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"]
Oct 30 19:19:04.928085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.927084    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.928085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.927157    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.928085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.927192    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467237f-3ab9-424a-9899-37f77850510a-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.928085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.927247    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqlp\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-kube-api-access-zvqlp\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.928085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.927297    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.928085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.927319    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467237f-3ab9-424a-9899-37f77850510a-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.928085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.927393    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.928085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.927439    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.928085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.927473    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:04.931339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.931080    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"]
Oct 30 19:19:04.934308 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:04.934265    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7327b5_03ab_49fa_89e7_08df93bf233d.slice/crio-463a82eb76f340a947c74160ebcf63b10511651e8344ada10e5f23e480d09690 WatchSource:0}: Error finding container 463a82eb76f340a947c74160ebcf63b10511651e8344ada10e5f23e480d09690: Status 404 returned error can't find the container with id 463a82eb76f340a947c74160ebcf63b10511651e8344ada10e5f23e480d09690
Oct 30 19:19:04.936883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.936844    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:04.936964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.936934    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:04.937006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.936978    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:04.969883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:04.969859    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:05.012214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.012189    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"]
Oct 30 19:19:05.014233 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:05.014207    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a0608a_109b_4aef_88ab_52b53be4ca45.slice/crio-d157cf5285e642a298c3f1de3ce3d48a837e647881db0a7e57dc9550f5d25b5d WatchSource:0}: Error finding container d157cf5285e642a298c3f1de3ce3d48a837e647881db0a7e57dc9550f5d25b5d: Status 404 returned error can't find the container with id d157cf5285e642a298c3f1de3ce3d48a837e647881db0a7e57dc9550f5d25b5d
Oct 30 19:19:05.016722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.016688    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:05.016835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.016772    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:05.016835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.016820    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.029277    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.029336    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.029368    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467237f-3ab9-424a-9899-37f77850510a-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.029406    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqlp\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-kube-api-access-zvqlp\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.029451    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.029475    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467237f-3ab9-424a-9899-37f77850510a-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.029549    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.029586    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.029618    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.030006    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.031549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.030278    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.032511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.032399    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467237f-3ab9-424a-9899-37f77850510a-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.032957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.032652    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.033363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.033342    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.038184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.038131    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.039001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.038795    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467237f-3ab9-424a-9899-37f77850510a-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.051895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.051839    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.056523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.056473    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqlp\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-kube-api-access-zvqlp\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.168465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.168185    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:05.269262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.269225    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"]
Oct 30 19:19:05.279361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.276946    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:05.279361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.277036    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:05.279361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.277098    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:05.397513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.397473    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" event={"ID":"fb7327b5-03ab-49fa-89e7-08df93bf233d","Type":"ContainerStarted","Data":"a765f43cfc2b6c6774889b9623c1125ec653f20553ba0519cfe955df3aa961c0"}
Oct 30 19:19:05.398084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.398044    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" event={"ID":"fb7327b5-03ab-49fa-89e7-08df93bf233d","Type":"ContainerStarted","Data":"463a82eb76f340a947c74160ebcf63b10511651e8344ada10e5f23e480d09690"}
Oct 30 19:19:05.399396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.399373    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" event={"ID":"16a0608a-109b-4aef-88ab-52b53be4ca45","Type":"ContainerStarted","Data":"6f3ba36c1d6b8eab6d00285847848023a7d77100a36f1e4090a27b88a6b7d0b4"}
Oct 30 19:19:05.399503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.399410    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" event={"ID":"16a0608a-109b-4aef-88ab-52b53be4ca45","Type":"ContainerStarted","Data":"d157cf5285e642a298c3f1de3ce3d48a837e647881db0a7e57dc9550f5d25b5d"}
Oct 30 19:19:05.401716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.401692    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" event={"ID":"e42325e6-a838-4786-9c57-08fa864e811a","Type":"ContainerStarted","Data":"6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562"}
Oct 30 19:19:05.401825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.401735    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" event={"ID":"e42325e6-a838-4786-9c57-08fa864e811a","Type":"ContainerStarted","Data":"77a334cbf054c42a5bf3d46700953008a34d7eb960196528124a22d0cd9f2867"}
Oct 30 19:19:05.423495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.423447    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" podStartSLOduration=1.423432843 podStartE2EDuration="1.423432843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:05.420374324 +0000 UTC m=+727.417289709" watchObservedRunningTime="2026-10-30 19:19:05.423432843 +0000 UTC m=+727.420348228"
Oct 30 19:19:05.454268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.454206    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" podStartSLOduration=1.454189035 podStartE2EDuration="1.454189035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:05.453122453 +0000 UTC m=+727.450037838" watchObservedRunningTime="2026-10-30 19:19:05.454189035 +0000 UTC m=+727.451104421"
Oct 30 19:19:05.472526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.472486    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"]
Oct 30 19:19:05.473809 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:05.473776    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2467237f_3ab9_424a_9899_37f77850510a.slice/crio-485252d31e84a73b99ecc74a004c408c9ba55c3318ac9b47e704455f5ddb50bc WatchSource:0}: Error finding container 485252d31e84a73b99ecc74a004c408c9ba55c3318ac9b47e704455f5ddb50bc: Status 404 returned error can't find the container with id 485252d31e84a73b99ecc74a004c408c9ba55c3318ac9b47e704455f5ddb50bc
Oct 30 19:19:05.487540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.476753    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:05.487540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.476916    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:05.487540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.477002    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:05.488964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.488905    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podStartSLOduration=1.48889256 podStartE2EDuration="1.48889256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:05.488380906 +0000 UTC m=+727.485296337" watchObservedRunningTime="2026-10-30 19:19:05.48889256 +0000 UTC m=+727.485807943"
Oct 30 19:19:05.625782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.625744    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:05.725258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.725226    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:05.970715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:05.970637    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:06.408727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:06.408689    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" event={"ID":"2467237f-3ab9-424a-9899-37f77850510a","Type":"ContainerStarted","Data":"fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c"}
Oct 30 19:19:06.409159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:06.408749    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" event={"ID":"2467237f-3ab9-424a-9899-37f77850510a","Type":"ContainerStarted","Data":"485252d31e84a73b99ecc74a004c408c9ba55c3318ac9b47e704455f5ddb50bc"}
Oct 30 19:19:06.437165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:06.437117    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" podStartSLOduration=2.437103866 podStartE2EDuration="2.437103866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:06.435743507 +0000 UTC m=+728.432658892" watchObservedRunningTime="2026-10-30 19:19:06.437103866 +0000 UTC m=+728.434019250"
Oct 30 19:19:06.631490 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:06.631459    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:06.730300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:06.730224    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:06.975413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:06.975383    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:07.169004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.168974    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:07.174220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.174193    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:07.413843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.413813    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:07.414299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.413866    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:07.414299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.413889    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:07.414299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.413911    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:07.414563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.414536    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:07.414700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.414658    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:07.414760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.414725    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:07.414879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:07.414867    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:11.682924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:11.682842    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:19:12.061981 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.061949    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"]
Oct 30 19:19:12.065920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.065903    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:12.086724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.086688    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"]
Oct 30 19:19:12.131397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.131367    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-bce71f-dockercfg-77zgh\""
Oct 30 19:19:12.132985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.132941    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-bce71f\""
Oct 30 19:19:12.223169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.223137    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfxrw\" (UniqueName: \"kubernetes.io/projected/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-kube-api-access-vfxrw\") pod \"payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9\" (UID: \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:12.223326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.223201    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9\" (UID: \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:12.324531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.324437    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfxrw\" (UniqueName: \"kubernetes.io/projected/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-kube-api-access-vfxrw\") pod \"payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9\" (UID: \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:12.324531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.324503    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9\" (UID: \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:12.325163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.325142    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9\" (UID: \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:12.337079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.337042    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfxrw\" (UniqueName: \"kubernetes.io/projected/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-kube-api-access-vfxrw\") pod \"payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9\" (UID: \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:12.379045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.378990    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:12.675677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.675540    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"]
Oct 30 19:19:12.678470 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:12.678441    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1299b4be_5ada_45f6_b7bb_9e0e25e6854f.slice/crio-884f58e79e9fe15764d7b9a15a3b68df813aa3964368ea9d8733fcf5736f3a5e WatchSource:0}: Error finding container 884f58e79e9fe15764d7b9a15a3b68df813aa3964368ea9d8733fcf5736f3a5e: Status 404 returned error can't find the container with id 884f58e79e9fe15764d7b9a15a3b68df813aa3964368ea9d8733fcf5736f3a5e
Oct 30 19:19:12.726464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.726432    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"]
Oct 30 19:19:12.731865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.731843    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:12.738552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.738525    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"]
Oct 30 19:19:12.740420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.739182    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ad23c7-dockercfg-thxlb\""
Oct 30 19:19:12.830965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.830932    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ad23c7\""
Oct 30 19:19:12.831440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.831415    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdg2g\" (UniqueName: \"kubernetes.io/projected/1f430439-89da-4b5a-9c8e-4bb5997ad43d-kube-api-access-mdg2g\") pod \"payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896\" (UID: \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:12.831529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.831486    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f430439-89da-4b5a-9c8e-4bb5997ad43d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896\" (UID: \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:12.933263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.933170    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdg2g\" (UniqueName: \"kubernetes.io/projected/1f430439-89da-4b5a-9c8e-4bb5997ad43d-kube-api-access-mdg2g\") pod \"payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896\" (UID: \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:12.933263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.933233    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f430439-89da-4b5a-9c8e-4bb5997ad43d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896\" (UID: \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:12.934231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.933937    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f430439-89da-4b5a-9c8e-4bb5997ad43d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896\" (UID: \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:12.946690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:12.946656    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdg2g\" (UniqueName: \"kubernetes.io/projected/1f430439-89da-4b5a-9c8e-4bb5997ad43d-kube-api-access-mdg2g\") pod \"payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896\" (UID: \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:13.047420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.047391    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:13.371083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.371025    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"]
Oct 30 19:19:13.380625 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:13.380580    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f430439_89da_4b5a_9c8e_4bb5997ad43d.slice/crio-1f3a579a24aeb5b2a091abffc3966664956314ecb3d95d3a1a617018cc860ad0 WatchSource:0}: Error finding container 1f3a579a24aeb5b2a091abffc3966664956314ecb3d95d3a1a617018cc860ad0: Status 404 returned error can't find the container with id 1f3a579a24aeb5b2a091abffc3966664956314ecb3d95d3a1a617018cc860ad0
Oct 30 19:19:13.498738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.498707    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896" event={"ID":"1f430439-89da-4b5a-9c8e-4bb5997ad43d","Type":"ContainerStarted","Data":"c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2"}
Oct 30 19:19:13.498898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.498773    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896" event={"ID":"1f430439-89da-4b5a-9c8e-4bb5997ad43d","Type":"ContainerStarted","Data":"1f3a579a24aeb5b2a091abffc3966664956314ecb3d95d3a1a617018cc860ad0"}
Oct 30 19:19:13.498982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.498918    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:13.500359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.500337    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9" event={"ID":"1299b4be-5ada-45f6-b7bb-9e0e25e6854f","Type":"ContainerStarted","Data":"0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81"}
Oct 30 19:19:13.500456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.500372    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9" event={"ID":"1299b4be-5ada-45f6-b7bb-9e0e25e6854f","Type":"ContainerStarted","Data":"884f58e79e9fe15764d7b9a15a3b68df813aa3964368ea9d8733fcf5736f3a5e"}
Oct 30 19:19:13.500456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.500397    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:13.519636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.519589    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896" podStartSLOduration=1.519572724 podStartE2EDuration="1.519572724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:13.519544815 +0000 UTC m=+735.516460202" watchObservedRunningTime="2026-10-30 19:19:13.519572724 +0000 UTC m=+735.516488110"
Oct 30 19:19:13.547170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.547124    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9" podStartSLOduration=1.547112237 podStartE2EDuration="1.547112237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:13.546901633 +0000 UTC m=+735.543817018" watchObservedRunningTime="2026-10-30 19:19:13.547112237 +0000 UTC m=+735.544027621"
Oct 30 19:19:13.644603 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:19:13.644511    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:19:13.737973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.737932    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"]
Oct 30 19:19:13.768083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.744240    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:13.768083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.746041    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"]
Oct 30 19:19:13.768083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.753957    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-39b867-dockercfg-556l9\""
Oct 30 19:19:13.832686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.832656    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-39b867\""
Oct 30 19:19:13.843173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.843138    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqt88\" (UniqueName: \"kubernetes.io/projected/52606f4d-8ec1-42d9-b49a-67218a118f6f-kube-api-access-pqt88\") pod \"payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6\" (UID: \"52606f4d-8ec1-42d9-b49a-67218a118f6f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:13.843322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.843279    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606f4d-8ec1-42d9-b49a-67218a118f6f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6\" (UID: \"52606f4d-8ec1-42d9-b49a-67218a118f6f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:13.944053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.943946    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqt88\" (UniqueName: \"kubernetes.io/projected/52606f4d-8ec1-42d9-b49a-67218a118f6f-kube-api-access-pqt88\") pod \"payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6\" (UID: \"52606f4d-8ec1-42d9-b49a-67218a118f6f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:13.944053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.944050    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606f4d-8ec1-42d9-b49a-67218a118f6f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6\" (UID: \"52606f4d-8ec1-42d9-b49a-67218a118f6f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:13.944752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.944727    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606f4d-8ec1-42d9-b49a-67218a118f6f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6\" (UID: \"52606f4d-8ec1-42d9-b49a-67218a118f6f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:13.981081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:13.977898    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqt88\" (UniqueName: \"kubernetes.io/projected/52606f4d-8ec1-42d9-b49a-67218a118f6f-kube-api-access-pqt88\") pod \"payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6\" (UID: \"52606f4d-8ec1-42d9-b49a-67218a118f6f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:14.114204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:14.114173    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:14.417524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:14.417490    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"]
Oct 30 19:19:14.418903 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:14.418876    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52606f4d_8ec1_42d9_b49a_67218a118f6f.slice/crio-7e4229ddf26c4a997d92e5636074336ca66cb176f1b72b3dec079213c4238943 WatchSource:0}: Error finding container 7e4229ddf26c4a997d92e5636074336ca66cb176f1b72b3dec079213c4238943: Status 404 returned error can't find the container with id 7e4229ddf26c4a997d92e5636074336ca66cb176f1b72b3dec079213c4238943
Oct 30 19:19:14.506974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:14.506931    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6" event={"ID":"52606f4d-8ec1-42d9-b49a-67218a118f6f","Type":"ContainerStarted","Data":"7e4229ddf26c4a997d92e5636074336ca66cb176f1b72b3dec079213c4238943"}
Oct 30 19:19:15.514905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:15.514847    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6" event={"ID":"52606f4d-8ec1-42d9-b49a-67218a118f6f","Type":"ContainerStarted","Data":"e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b"}
Oct 30 19:19:15.515644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:15.515605    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:15.537734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:15.537670    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6" podStartSLOduration=2.537651081 podStartE2EDuration="2.537651081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:15.536717482 +0000 UTC m=+737.533632867" watchObservedRunningTime="2026-10-30 19:19:15.537651081 +0000 UTC m=+737.534566466"
Oct 30 19:19:17.882964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:17.882925    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"]
Oct 30 19:19:17.887620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:17.887599    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:17.899676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:17.898976    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:19:17.899676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:17.899166    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:19:17.899676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:17.899252    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:19:17.900261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:17.900179    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f8msk\""
Oct 30 19:19:17.900261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:17.900223    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ad23c7-serving-cert\""
Oct 30 19:19:17.901530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:17.901503    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"]
Oct 30 19:19:18.022838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.022807    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c315da14-f28a-4be1-a90d-ff186090fb9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.022985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.022855    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.022985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.022958    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkz4\" (UniqueName: \"kubernetes.io/projected/c315da14-f28a-4be1-a90d-ff186090fb9e-kube-api-access-dbkz4\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.023238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.023187    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.124649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.124614    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.124808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.124706    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c315da14-f28a-4be1-a90d-ff186090fb9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.124808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.124756    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.124928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.124810    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkz4\" (UniqueName: \"kubernetes.io/projected/c315da14-f28a-4be1-a90d-ff186090fb9e-kube-api-access-dbkz4\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.124928 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:19:18.124757    6777 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-serving-cert: secret "maas-api-e2e-worker-w0-ad23c7-serving-cert" not found
Oct 30 19:19:18.125003 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:19:18.124927    6777 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls podName:c315da14-f28a-4be1-a90d-ff186090fb9e nodeName:}" failed. No retries permitted until 2026-10-30 19:19:18.624899891 +0000 UTC m=+740.621815264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls") pod "maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc" (UID: "c315da14-f28a-4be1-a90d-ff186090fb9e") : secret "maas-api-e2e-worker-w0-ad23c7-serving-cert" not found
Oct 30 19:19:18.125439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.125414    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c315da14-f28a-4be1-a90d-ff186090fb9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.127288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.127260    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.138529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.138468    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkz4\" (UniqueName: \"kubernetes.io/projected/c315da14-f28a-4be1-a90d-ff186090fb9e-kube-api-access-dbkz4\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.630426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.630383    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.632860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.632835    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:18.808199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:18.808159    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:19.109896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:19.109869    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"]
Oct 30 19:19:19.111938 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:19.111911    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc315da14_f28a_4be1_a90d_ff186090fb9e.slice/crio-471b155faf8ce189a3caeb115d62b2a04307bc5ea2ee84d2133e2be6dbde4dba WatchSource:0}: Error finding container 471b155faf8ce189a3caeb115d62b2a04307bc5ea2ee84d2133e2be6dbde4dba: Status 404 returned error can't find the container with id 471b155faf8ce189a3caeb115d62b2a04307bc5ea2ee84d2133e2be6dbde4dba
Oct 30 19:19:19.538768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:19.538726    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc" event={"ID":"c315da14-f28a-4be1-a90d-ff186090fb9e","Type":"ContainerStarted","Data":"471b155faf8ce189a3caeb115d62b2a04307bc5ea2ee84d2133e2be6dbde4dba"}
Oct 30 19:19:20.903851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:20.903371    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:19:22.958292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:22.946355    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:58544->10.134.0.40:15021: read: connection reset by peer" start-of-body=
Oct 30 19:19:22.958292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:22.946449    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:58544->10.134.0.40:15021: read: connection reset by peer"
Oct 30 19:19:23.478084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:23.462289    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:19:23.478084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:23.462354    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:19:23.612427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:23.612391    6777 generic.go:356] "Generic (PLEG): container finished" podID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerID="6f3ba36c1d6b8eab6d00285847848023a7d77100a36f1e4090a27b88a6b7d0b4" exitCode=0
Oct 30 19:19:23.612556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:23.612455    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" event={"ID":"16a0608a-109b-4aef-88ab-52b53be4ca45","Type":"ContainerDied","Data":"6f3ba36c1d6b8eab6d00285847848023a7d77100a36f1e4090a27b88a6b7d0b4"}
Oct 30 19:19:23.613071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:23.613028    6777 scope.go:122] "RemoveContainer" containerID="6f3ba36c1d6b8eab6d00285847848023a7d77100a36f1e4090a27b88a6b7d0b4"
Oct 30 19:19:23.614834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:23.614485    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:23.614834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:23.614552    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:23.614834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:23.614588    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:24.541662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:24.541618    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:19:24.543241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:24.543215    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:19:24.637345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:24.637308    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" event={"ID":"16a0608a-109b-4aef-88ab-52b53be4ca45","Type":"ContainerStarted","Data":"d87dd1c13722607b0c066f4101923be25b61449ad51178f3f1210fe8364a7f1f"}
Oct 30 19:19:24.640752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:24.639369    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc" event={"ID":"c315da14-f28a-4be1-a90d-ff186090fb9e","Type":"ContainerStarted","Data":"1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f"}
Oct 30 19:19:24.640752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:24.639744    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:24.736864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:24.736838    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:24.737831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:24.737811    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:19:24.738286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:24.738262    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:19:24.786083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:24.761770    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc" podStartSLOduration=3.306011648 podStartE2EDuration="7.761753545s" totalImagesPullingTime="4.455741897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:24.736464531 +0000 UTC m=+746.733379915" watchObservedRunningTime="2026-10-30 19:19:24.761753545 +0000 UTC m=+746.758668931"
Oct 30 19:19:25.727611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:25.727566    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:19:25.728317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:25.728263    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:19:26.686309 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:26.686268    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:19:26.727601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:26.727568    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:19:26.727903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:26.727855    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:19:27.536036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:27.536000    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:19:27.730081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:27.729159    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:19:27.730081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:27.729221    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:19:28.726040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:28.725496    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:19:28.726040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:28.725564    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:19:29.725779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:29.725289    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:19:29.725779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:29.725358    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:19:30.626600    6777 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42325e6_a838_4786_9c57_08fa864e811a.slice/crio-6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@1-14282-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:30.729240    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:30.729544    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:30.730257    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:30.755347    6777 generic.go:356] "Generic (PLEG): container finished" podID="e42325e6-a838-4786-9c57-08fa864e811a" containerID="6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562" exitCode=0
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:30.755427    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" event={"ID":"e42325e6-a838-4786-9c57-08fa864e811a","Type":"ContainerDied","Data":"6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562"}
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:30.755847    6777 scope.go:122] "RemoveContainer" containerID="6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562"
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:30.760510    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:30.760592    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:31.483775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:30.760642    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:31.682162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:31.682131    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:19:31.771242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:31.771148    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" event={"ID":"e42325e6-a838-4786-9c57-08fa864e811a","Type":"ContainerStarted","Data":"8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5"}
Oct 30 19:19:31.975085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:31.974786    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:31.975085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:31.974911    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:31.975085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:31.974957    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:19:32.977968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:32.977507    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:32.977968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:32.977578    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:19:33.785158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:33.785122    6777 generic.go:356] "Generic (PLEG): container finished" podID="2467237f-3ab9-424a-9899-37f77850510a" containerID="fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c" exitCode=0
Oct 30 19:19:33.800758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:33.785198    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" event={"ID":"2467237f-3ab9-424a-9899-37f77850510a","Type":"ContainerDied","Data":"fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c"}
Oct 30 19:19:33.800758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:33.785615    6777 scope.go:122] "RemoveContainer" containerID="fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c"
Oct 30 19:19:33.800758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:33.786966    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:33.800758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:33.787081    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:33.800758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:33.787114    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:33.973030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:33.971337    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:33.973030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:33.971448    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:19:34.803818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:34.803774    6777 generic.go:356] "Generic (PLEG): container finished" podID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerID="a765f43cfc2b6c6774889b9623c1125ec653f20553ba0519cfe955df3aa961c0" exitCode=0
Oct 30 19:19:34.804325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:34.803846    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" event={"ID":"fb7327b5-03ab-49fa-89e7-08df93bf233d","Type":"ContainerDied","Data":"a765f43cfc2b6c6774889b9623c1125ec653f20553ba0519cfe955df3aa961c0"}
Oct 30 19:19:34.804325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:34.804279    6777 scope.go:122] "RemoveContainer" containerID="a765f43cfc2b6c6774889b9623c1125ec653f20553ba0519cfe955df3aa961c0"
Oct 30 19:19:34.805521 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:34.805490    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:34.805703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:34.805689    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:34.805791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:34.805780    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:19:34.806099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:34.806077    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" event={"ID":"2467237f-3ab9-424a-9899-37f77850510a","Type":"ContainerStarted","Data":"1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486"}
Oct 30 19:19:34.992433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:34.991503    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:34.992433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:34.991567    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:19:35.178142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:35.176442    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:35.186079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:35.185204    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:35.186079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:35.185286    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:19:35.828448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:35.828414    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" event={"ID":"fb7327b5-03ab-49fa-89e7-08df93bf233d","Type":"ContainerStarted","Data":"7b5e16a3718a23d71b1abff2557a8e02da2355d2075c9216d0b3f95edd55371a"}
Oct 30 19:19:35.973084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:35.972394    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:35.973084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:35.972457    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:19:36.177040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:36.176516    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:36.177040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:36.176585    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:19:36.628171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:36.628134    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:36.629083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:36.628914    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:36.629083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:36.628972    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 19:19:36.839551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:36.839465    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:19:36.985138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:36.984465    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:36.985138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:36.984534    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:19:37.176720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:37.176686    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:37.177012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:37.176980    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:19:37.415256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:37.415219    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:37.416301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:37.416269    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:37.416761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:37.416741    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:37.625810 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:37.625771    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:37.625979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:37.625836    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 19:19:37.993691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:37.993659    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:38.022209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:38.022169    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:19:38.187083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:38.183530    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:38.187083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:38.183596    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:19:38.625525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:38.625493    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:38.625741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:38.625719    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 19:19:39.174513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:39.174476    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:39.175177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:39.175140    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:19:39.626159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:39.626123    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:39.626385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:39.626186    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 19:19:40.199818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:40.199776    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:40.201210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:40.201173    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:19:40.626394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:40.626357    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:40.626576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:40.626414    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 19:19:41.678484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.678450    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:41.681771 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.681749    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:19:41.760710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.760679    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"]
Oct 30 19:19:41.763314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.763280    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.767985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.767949    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-17e904\"/\"openshift-service-ca.crt\""
Oct 30 19:19:41.768160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.768147    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17e904\"/\"default-dockercfg-khc9r\""
Oct 30 19:19:41.768218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.768195    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17e904\"/\"simulator-w3-17e904-kserve-self-signed-certs\""
Oct 30 19:19:41.770657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.770600    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-17e904\"/\"kube-root-ca.crt\""
Oct 30 19:19:41.773931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.773905    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"]
Oct 30 19:19:41.889263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.889225    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tmp-dir\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.889263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.889268    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tls-certs\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.889462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.889417    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmxb4\" (UniqueName: \"kubernetes.io/projected/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-kube-api-access-vmxb4\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.889462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.889445    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-model-cache\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.889561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.889546    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-home\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.889597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.889577    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-dshm\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.990417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.990314    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmxb4\" (UniqueName: \"kubernetes.io/projected/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-kube-api-access-vmxb4\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.990417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.990364    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-model-cache\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.990664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.990442    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-home\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.990664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.990475    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-dshm\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.990664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.990570    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tmp-dir\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.990664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.990632    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tls-certs\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.990863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.990823    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-model-cache\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.990928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.990868    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-home\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.991004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.990974    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tmp-dir\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.993092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.993049    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-dshm\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:41.993203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:41.993178    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tls-certs\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:42.006709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:42.006686    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmxb4\" (UniqueName: \"kubernetes.io/projected/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-kube-api-access-vmxb4\") pod \"simulator-w3-17e904-kserve-6688676f9b-ld4zl\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") " pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:42.078407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:42.078365    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:42.393426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:42.392301    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"]
Oct 30 19:19:42.398335 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:42.398308    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfe94c2e_56f0_4979_8b7c_d83baf87ea6a.slice/crio-b0bf33442d9293d6908554214693996a2f6cda50f2995d28fc3064ebddd5a2a6 WatchSource:0}: Error finding container b0bf33442d9293d6908554214693996a2f6cda50f2995d28fc3064ebddd5a2a6: Status 404 returned error can't find the container with id b0bf33442d9293d6908554214693996a2f6cda50f2995d28fc3064ebddd5a2a6
Oct 30 19:19:42.911303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:42.911265    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" event={"ID":"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a","Type":"ContainerStarted","Data":"ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f"}
Oct 30 19:19:42.911303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:42.911309    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" event={"ID":"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a","Type":"ContainerStarted","Data":"b0bf33442d9293d6908554214693996a2f6cda50f2995d28fc3064ebddd5a2a6"}
Oct 30 19:19:42.931400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:42.931350    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podStartSLOduration=1.931336257 podStartE2EDuration="1.931336257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:42.930603373 +0000 UTC m=+764.927518756" watchObservedRunningTime="2026-10-30 19:19:42.931336257 +0000 UTC m=+764.928251690"
Oct 30 19:19:43.079446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:43.079413    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:43.638491 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:19:43.638453    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:19:45.158737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:45.158703    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:19:46.487775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.487731    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"]
Oct 30 19:19:46.491717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.491697    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.497696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.497672    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bce71f\"/\"kube-root-ca.crt\""
Oct 30 19:19:46.499233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.499205    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bce71f\"/\"openshift-service-ca.crt\""
Oct 30 19:19:46.499356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.499285    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bce71f\"/\"default-dockercfg-rgxzp\""
Oct 30 19:19:46.499356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.499320    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bce71f\"/\"simulator-w6-bce71f-kserve-self-signed-certs\""
Oct 30 19:19:46.501681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.501656    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"]
Oct 30 19:19:46.540703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.540669    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tls-certs\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.541011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.540744    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tmp-dir\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.541011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.540840    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-dshm\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.541011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.540890    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-home\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.541011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.540913    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-model-cache\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.541011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.540938    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6l5\" (UniqueName: \"kubernetes.io/projected/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-kube-api-access-8g6l5\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.641955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.641921    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tls-certs\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.642187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.641988    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tmp-dir\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.642187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.642017    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-dshm\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.642187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.642085    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-home\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.642187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.642110    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-model-cache\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.642187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.642137    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6l5\" (UniqueName: \"kubernetes.io/projected/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-kube-api-access-8g6l5\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.642531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.642508    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tmp-dir\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.642617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.642559    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-model-cache\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.642617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.642590    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-home\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.644447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.644424    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-dshm\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.644669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.644650    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tls-certs\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.655891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.655860    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6l5\" (UniqueName: \"kubernetes.io/projected/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-kube-api-access-8g6l5\") pod \"simulator-w6-bce71f-kserve-69f47c5589-nbrzf\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") " pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:46.806212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:46.806177    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:47.086320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:47.086291    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"]
Oct 30 19:19:47.088283 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:47.088246    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a27f460_fdfb_49eb_a8ee_2894d9d885ad.slice/crio-5e601ce3222f9bbb1c3b559d3ce67502612528c945fc15a927ed2e0b2848ac2e WatchSource:0}: Error finding container 5e601ce3222f9bbb1c3b559d3ce67502612528c945fc15a927ed2e0b2848ac2e: Status 404 returned error can't find the container with id 5e601ce3222f9bbb1c3b559d3ce67502612528c945fc15a927ed2e0b2848ac2e
Oct 30 19:19:47.941792 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:47.941754    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" event={"ID":"8a27f460-fdfb-49eb-a8ee-2894d9d885ad","Type":"ContainerStarted","Data":"a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4"}
Oct 30 19:19:47.942366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:47.941803    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" event={"ID":"8a27f460-fdfb-49eb-a8ee-2894d9d885ad","Type":"ContainerStarted","Data":"5e601ce3222f9bbb1c3b559d3ce67502612528c945fc15a927ed2e0b2848ac2e"}
Oct 30 19:19:47.966035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:47.965983    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podStartSLOduration=1.965970675 podStartE2EDuration="1.965970675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:47.962144054 +0000 UTC m=+769.959059446" watchObservedRunningTime="2026-10-30 19:19:47.965970675 +0000 UTC m=+769.962886076"
Oct 30 19:19:48.806824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:48.806780    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:52.079165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.079130    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:52.091724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.091696    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:52.254050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.254012    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"]
Oct 30 19:19:52.258396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.258369    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.267012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.266976    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"]
Oct 30 19:19:52.268384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.268362    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"kube-root-ca.crt\""
Oct 30 19:19:52.268501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.268413    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"openshift-service-ca.crt\""
Oct 30 19:19:52.268632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.268611    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"default-dockercfg-fq74v\""
Oct 30 19:19:52.268700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.268676    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"premium-w1-cfa4bb-kserve-self-signed-certs\""
Oct 30 19:19:52.410309 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.410220    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e3c06-7423-46e4-b910-bba1a9147b70-tls-certs\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.410464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.410368    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-tmp-dir\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.410464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.410395    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-model-cache\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.410464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.410450    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-home\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.410557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.410525    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-dshm\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.410591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.410574    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66dkb\" (UniqueName: \"kubernetes.io/projected/3c9e3c06-7423-46e4-b910-bba1a9147b70-kube-api-access-66dkb\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.511449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.511413    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66dkb\" (UniqueName: \"kubernetes.io/projected/3c9e3c06-7423-46e4-b910-bba1a9147b70-kube-api-access-66dkb\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.511630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.511466    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e3c06-7423-46e4-b910-bba1a9147b70-tls-certs\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.511630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.511538    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-tmp-dir\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.511630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.511554    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-model-cache\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.511791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.511666    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-home\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.511791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.511749    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-dshm\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.511948 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.511931    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-tmp-dir\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.512049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.512025    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-model-cache\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.512115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.512031    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-home\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.513858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.513837    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-dshm\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.514221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.514201    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e3c06-7423-46e4-b910-bba1a9147b70-tls-certs\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.524663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.524640    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66dkb\" (UniqueName: \"kubernetes.io/projected/3c9e3c06-7423-46e4-b910-bba1a9147b70-kube-api-access-66dkb\") pod \"premium-w1-cfa4bb-kserve-8f77fb569-zlzbp\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") " pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.573959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.573927    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:52.849600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.849573    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"]
Oct 30 19:19:52.851120 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:19:52.851093    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c9e3c06_7423_46e4_b910_bba1a9147b70.slice/crio-2002d35d0b24ead4754f0f7692fc3c0560311d1b58e23e466758606eaa0b877f WatchSource:0}: Error finding container 2002d35d0b24ead4754f0f7692fc3c0560311d1b58e23e466758606eaa0b877f: Status 404 returned error can't find the container with id 2002d35d0b24ead4754f0f7692fc3c0560311d1b58e23e466758606eaa0b877f
Oct 30 19:19:52.972614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.972573    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" event={"ID":"3c9e3c06-7423-46e4-b910-bba1a9147b70","Type":"ContainerStarted","Data":"4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c"}
Oct 30 19:19:52.972784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.972629    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" event={"ID":"3c9e3c06-7423-46e4-b910-bba1a9147b70","Type":"ContainerStarted","Data":"2002d35d0b24ead4754f0f7692fc3c0560311d1b58e23e466758606eaa0b877f"}
Oct 30 19:19:52.984249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.984212    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:19:52.994319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:52.994277    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podStartSLOduration=0.994265849 podStartE2EDuration="994.265849ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:52.991584145 +0000 UTC m=+774.988499531" watchObservedRunningTime="2026-10-30 19:19:52.994265849 +0000 UTC m=+774.991181272"
Oct 30 19:19:53.574474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:53.574440    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:19:55.797381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:55.797341    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:19:56.807151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:56.807111    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:56.824413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:56.824385    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:19:57.017819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:19:57.017783    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:20:02.574667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:02.574632    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:20:02.587000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:02.586968    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:20:03.043269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:03.043241    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:20:06.986478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:06.986447    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"]
Oct 30 19:20:06.991608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:06.991585    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:06.999909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:06.999878    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"]
Oct 30 19:20:07.000813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.000791    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-27b5d5\"/\"openshift-service-ca.crt\""
Oct 30 19:20:07.002334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.002310    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-27b5d5\"/\"kube-root-ca.crt\""
Oct 30 19:20:07.002460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.002345    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-27b5d5\"/\"premium-w4-27b5d5-kserve-self-signed-certs\""
Oct 30 19:20:07.002460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.002425    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-27b5d5\"/\"default-dockercfg-x4446\""
Oct 30 19:20:07.124768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.124724    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-home\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.125089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.124779    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tls-certs\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.125089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.124839    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt96j\" (UniqueName: \"kubernetes.io/projected/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-kube-api-access-lt96j\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.125089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.124881    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-dshm\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.125089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.125007    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-model-cache\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.125089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.125047    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tmp-dir\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.225756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.225716    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-model-cache\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.225946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.225773    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tmp-dir\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.225946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.225806    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-home\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.225946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.225826    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tls-certs\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.226171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.225970    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt96j\" (UniqueName: \"kubernetes.io/projected/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-kube-api-access-lt96j\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.226171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.226041    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-dshm\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.226171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.226158    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-model-cache\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.226310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.226192    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-home\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.226310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.226256    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tmp-dir\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.228183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.228158    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-dshm\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.228385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.228369    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tls-certs\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.238538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.238488    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt96j\" (UniqueName: \"kubernetes.io/projected/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-kube-api-access-lt96j\") pod \"premium-w4-27b5d5-kserve-b44b4c977-tlk5f\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") " pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.306149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.306121    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:07.582389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:07.582354    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"]
Oct 30 19:20:07.583722 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:20:07.583690    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe6fc38a_2eaf_4d7b_8f86_1a983d77fcbd.slice/crio-20f21135b7db5558afd1acebc3b980d274356cc004c9f74ffb0f8816012e9f51 WatchSource:0}: Error finding container 20f21135b7db5558afd1acebc3b980d274356cc004c9f74ffb0f8816012e9f51: Status 404 returned error can't find the container with id 20f21135b7db5558afd1acebc3b980d274356cc004c9f74ffb0f8816012e9f51
Oct 30 19:20:08.061978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:08.061944    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" event={"ID":"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd","Type":"ContainerStarted","Data":"c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93"}
Oct 30 19:20:08.062373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:08.061990    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" event={"ID":"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd","Type":"ContainerStarted","Data":"20f21135b7db5558afd1acebc3b980d274356cc004c9f74ffb0f8816012e9f51"}
Oct 30 19:20:08.083423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:08.083370    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podStartSLOduration=2.083357067 podStartE2EDuration="2.083357067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:08.081656712 +0000 UTC m=+790.078572099" watchObservedRunningTime="2026-10-30 19:20:08.083357067 +0000 UTC m=+790.080272452"
Oct 30 19:20:08.306456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:08.306426    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:10.076128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.076092    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"]
Oct 30 19:20:10.080781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.080757    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.097973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.097925    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17e904\"/\"premium-w3-17e904-kserve-self-signed-certs\""
Oct 30 19:20:10.098485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.098465    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"]
Oct 30 19:20:10.157816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.157783    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-model-cache\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.157972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.157827    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-dshm\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.158018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.157972    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-tmp-dir\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.158096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.158050    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95t5\" (UniqueName: \"kubernetes.io/projected/3ff91e8e-7ada-4741-815e-06c9955b4151-kube-api-access-x95t5\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.158213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.158197    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-home\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.158249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.158237    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff91e8e-7ada-4741-815e-06c9955b4151-tls-certs\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.258935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.258899    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-tmp-dir\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.259188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.258964    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95t5\" (UniqueName: \"kubernetes.io/projected/3ff91e8e-7ada-4741-815e-06c9955b4151-kube-api-access-x95t5\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.259188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.259091    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-home\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.259188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.259124    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff91e8e-7ada-4741-815e-06c9955b4151-tls-certs\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.259188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.259151    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-model-cache\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.259188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.259184    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-dshm\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.259471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.259443    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-tmp-dir\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.259529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.259457    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-home\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.259574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.259551    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-model-cache\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.261367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.261339    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-dshm\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.261646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.261630    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff91e8e-7ada-4741-815e-06c9955b4151-tls-certs\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.271677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.271654    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95t5\" (UniqueName: \"kubernetes.io/projected/3ff91e8e-7ada-4741-815e-06c9955b4151-kube-api-access-x95t5\") pod \"premium-w3-17e904-kserve-5cf57bbb69-77lzf\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") " pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.358651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.358570    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:20:10.400175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.400135    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:10.705437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:10.705379    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"]
Oct 30 19:20:11.082436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:11.082389    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" event={"ID":"3ff91e8e-7ada-4741-815e-06c9955b4151","Type":"ContainerStarted","Data":"a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f"}
Oct 30 19:20:11.082912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:11.082461    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" event={"ID":"3ff91e8e-7ada-4741-815e-06c9955b4151","Type":"ContainerStarted","Data":"fcfcf69b553be810ccef5f714ea2d5ebd14b7a8aa9fe3ff42e81506f4bcb4b2d"}
Oct 30 19:20:11.104789 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:11.104737    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podStartSLOduration=1.104723021 podStartE2EDuration="1.104723021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:11.102655467 +0000 UTC m=+793.099570851" watchObservedRunningTime="2026-10-30 19:20:11.104723021 +0000 UTC m=+793.101638455"
Oct 30 19:20:11.401167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:11.401052    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:13.638594 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:20:13.638556    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:20:17.306871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:17.306829    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:17.320103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:17.320074    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:18.132202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:18.132167    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:20:19.678222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.678173    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"]
Oct 30 19:20:19.691811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.691779    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.699100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.698181    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-bce71f-openshift-default-dockercfg-w6thn\""
Oct 30 19:20:19.699939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.699913    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"]
Oct 30 19:20:19.884711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.884666    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.884711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.884708    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.884977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.884728    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a0e32e-00a4-4186-b20b-da060900cc17-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.884977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.884747    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.884977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.884826    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.884977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.884862    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.884977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.884960    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66tt\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-kube-api-access-p66tt\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.885273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.885045    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.885273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.885145    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.989864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.989776    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.990133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.990096    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.990280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.990265    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a0e32e-00a4-4186-b20b-da060900cc17-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.990380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.990368    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.990478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.990464    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.990574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.990562    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.990695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.990682    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66tt\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-kube-api-access-p66tt\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.990813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.990802    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.990944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.990932    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.991605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.991583    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.991977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.991960    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.992396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.992332    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.998077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.993081    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a0e32e-00a4-4186-b20b-da060900cc17-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:19.998239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:19.995367    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:20.006454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.006414    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:20.021101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.019648    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:20.024637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.024606    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:20.024762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.024608    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66tt\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-kube-api-access-p66tt\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:20.312149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.312115    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:20.418090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.416588    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:20.479550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.478714    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:20.677148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.676892    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"]
Oct 30 19:20:20.696623 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:20:20.696584    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a0e32e_00a4_4186_b20b_da060900cc17.slice/crio-5b634be6685b677780d05eaeea0319b873c14d40341f966134e2c9c8c2ce0b17 WatchSource:0}: Error finding container 5b634be6685b677780d05eaeea0319b873c14d40341f966134e2c9c8c2ce0b17: Status 404 returned error can't find the container with id 5b634be6685b677780d05eaeea0319b873c14d40341f966134e2c9c8c2ce0b17
Oct 30 19:20:20.699163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.699128    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:20:20.699280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.699202    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:20:20.699280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:20.699231    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:20:21.167630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:21.167580    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" event={"ID":"b7a0e32e-00a4-4186-b20b-da060900cc17","Type":"ContainerStarted","Data":"cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713"}
Oct 30 19:20:21.167825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:21.167646    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" event={"ID":"b7a0e32e-00a4-4186-b20b-da060900cc17","Type":"ContainerStarted","Data":"5b634be6685b677780d05eaeea0319b873c14d40341f966134e2c9c8c2ce0b17"}
Oct 30 19:20:21.179469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:21.179438    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:20:21.197309 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:21.197242    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" podStartSLOduration=2.197223864 podStartE2EDuration="2.197223864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:21.195529271 +0000 UTC m=+803.192444657" watchObservedRunningTime="2026-10-30 19:20:21.197223864 +0000 UTC m=+803.194139250"
Oct 30 19:20:21.313231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:21.313198    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:21.314761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:21.314713    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:20:21.314893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:21.314780    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:20:22.312576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:22.312545    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:20:22.312944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:22.312602    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:20:23.313401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:23.313361    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:20:23.313899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:23.313424    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:20:24.313183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:24.313139    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:20:24.313368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:24.313206    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:20:25.314986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:25.314551    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:20:25.314986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:25.314627    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:20:25.878774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:25.878729    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:20:26.183734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.181442    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"]
Oct 30 19:20:26.186787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.186758    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.205968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.205932    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"]
Oct 30 19:20:26.207630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.207604    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"openshift-service-ca.crt\""
Oct 30 19:20:26.207955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.207925    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"kube-root-ca.crt\""
Oct 30 19:20:26.208069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.207949    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"default-dockercfg-lxwmh\""
Oct 30 19:20:26.208069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.208035    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"premium-w0-ad23c7-kserve-self-signed-certs\""
Oct 30 19:20:26.300547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.300511    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-dshm\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.300547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.300548    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-model-cache\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.300778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.300569    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjm2n\" (UniqueName: \"kubernetes.io/projected/d5e318ad-55f2-4294-a8c5-10400322462b-kube-api-access-xjm2n\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.300778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.300740    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-tmp-dir\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.300841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.300816    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e318ad-55f2-4294-a8c5-10400322462b-tls-certs\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.300841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.300835    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-home\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.313070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.313031    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:20:26.313222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.313123    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:20:26.402386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.402342    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-dshm\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.402901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.402398    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-model-cache\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.402901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.402444    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjm2n\" (UniqueName: \"kubernetes.io/projected/d5e318ad-55f2-4294-a8c5-10400322462b-kube-api-access-xjm2n\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.403025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.402929    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-model-cache\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.403138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.403111    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-tmp-dir\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.403215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.403199    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e318ad-55f2-4294-a8c5-10400322462b-tls-certs\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.403377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.403347    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-home\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.403526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.403501    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-tmp-dir\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.403622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.403613    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-home\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.405818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.405792    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-dshm\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.406395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.406357    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e318ad-55f2-4294-a8c5-10400322462b-tls-certs\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.419557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.419514    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjm2n\" (UniqueName: \"kubernetes.io/projected/d5e318ad-55f2-4294-a8c5-10400322462b-kube-api-access-xjm2n\") pod \"premium-w0-ad23c7-kserve-959f5cf5d-qkzfx\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") " pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.505039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.504949    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:26.862043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.860954    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"]
Oct 30 19:20:26.927876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.927848    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"]
Oct 30 19:20:26.932209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.932183    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:26.951977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.948722    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:26.951977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.951855    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"]
Oct 30 19:20:26.953373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:26.953278    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:20:27.017366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.017331    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80302abc-qwjx9\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.017603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.017589    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vzz\" (UniqueName: \"kubernetes.io/projected/bec1b2e5-8104-403a-93b9-90bb0b3432f5-kube-api-access-d5vzz\") pod \"maas-api-revoke-sub-80302abc-qwjx9\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.017911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.017886    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80302abc-qwjx9\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.118890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.118798    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vzz\" (UniqueName: \"kubernetes.io/projected/bec1b2e5-8104-403a-93b9-90bb0b3432f5-kube-api-access-d5vzz\") pod \"maas-api-revoke-sub-80302abc-qwjx9\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.119094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.118941    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80302abc-qwjx9\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.119094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.118994    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80302abc-qwjx9\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.119633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.119614    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80302abc-qwjx9\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.121579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.121552    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80302abc-qwjx9\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.132814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.132787    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vzz\" (UniqueName: \"kubernetes.io/projected/bec1b2e5-8104-403a-93b9-90bb0b3432f5-kube-api-access-d5vzz\") pod \"maas-api-revoke-sub-80302abc-qwjx9\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.216822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.216784    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" event={"ID":"d5e318ad-55f2-4294-a8c5-10400322462b","Type":"ContainerStarted","Data":"27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6"}
Oct 30 19:20:27.217008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.216837    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" event={"ID":"d5e318ad-55f2-4294-a8c5-10400322462b","Type":"ContainerStarted","Data":"f128b3fba517388c63c209af9eec5dde5b4d3cea5e61b43d8a9d3fa5fe8fb05d"}
Oct 30 19:20:27.238463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.238414    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podStartSLOduration=1.238397696 podStartE2EDuration="1.238397696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:27.237014995 +0000 UTC m=+809.233930383" watchObservedRunningTime="2026-10-30 19:20:27.238397696 +0000 UTC m=+809.235313079"
Oct 30 19:20:27.259356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.259320    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:27.312809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.312775    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:20:27.313025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.312998    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:20:27.505613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.505513    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:27.541499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:27.541475    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"]
Oct 30 19:20:27.543268 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:20:27.543231    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec1b2e5_8104_403a_93b9_90bb0b3432f5.slice/crio-f47cd1a114bda53379c748d626cfc0d22e556bcb725721b28beb69fd76a93cb3 WatchSource:0}: Error finding container f47cd1a114bda53379c748d626cfc0d22e556bcb725721b28beb69fd76a93cb3: Status 404 returned error can't find the container with id f47cd1a114bda53379c748d626cfc0d22e556bcb725721b28beb69fd76a93cb3
Oct 30 19:20:28.223753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:28.223719    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9" event={"ID":"bec1b2e5-8104-403a-93b9-90bb0b3432f5","Type":"ContainerStarted","Data":"f47cd1a114bda53379c748d626cfc0d22e556bcb725721b28beb69fd76a93cb3"}
Oct 30 19:20:28.317994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:28.317957    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:28.318275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:28.318255    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:28.318895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:28.318878    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:20:31.246319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:31.246274    6777 generic.go:356] "Generic (PLEG): container finished" podID="bec1b2e5-8104-403a-93b9-90bb0b3432f5" containerID="5ce88a239c1ed29d92f5b12be6930fa3deeee67ffba5181c5351dead829a93c8" exitCode=0
Oct 30 19:20:31.246711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:31.246338    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9" event={"ID":"bec1b2e5-8104-403a-93b9-90bb0b3432f5","Type":"ContainerDied","Data":"5ce88a239c1ed29d92f5b12be6930fa3deeee67ffba5181c5351dead829a93c8"}
Oct 30 19:20:32.390849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.390823    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:32.486572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.486541    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-xks-ca\") pod \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") "
Oct 30 19:20:32.486760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.486590    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec1b2e5-8104-403a-93b9-90bb0b3432f5-kube-api-access-d5vzz\" (UniqueName: \"kubernetes.io/projected/bec1b2e5-8104-403a-93b9-90bb0b3432f5-kube-api-access-d5vzz\") pod \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") "
Oct 30 19:20:32.486760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.486668    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-openshift-ca\") pod \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\" (UID: \"bec1b2e5-8104-403a-93b9-90bb0b3432f5\") "
Oct 30 19:20:32.487199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.487095    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-openshift-ca" pod "bec1b2e5-8104-403a-93b9-90bb0b3432f5" (UID: "bec1b2e5-8104-403a-93b9-90bb0b3432f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:32.489024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.488998    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-xks-ca" pod "bec1b2e5-8104-403a-93b9-90bb0b3432f5" (UID: "bec1b2e5-8104-403a-93b9-90bb0b3432f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:32.489161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.489094    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec1b2e5-8104-403a-93b9-90bb0b3432f5-kube-api-access-d5vzz" pod "bec1b2e5-8104-403a-93b9-90bb0b3432f5" (UID: "bec1b2e5-8104-403a-93b9-90bb0b3432f5"). InnerVolumeSpecName "kube-api-access-d5vzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:32.588424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.588388    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.588424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.588418    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5vzz\" (UniqueName: \"kubernetes.io/projected/bec1b2e5-8104-403a-93b9-90bb0b3432f5-kube-api-access-d5vzz\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.588424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:32.588430    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1b2e5-8104-403a-93b9-90bb0b3432f5-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:33.258640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:33.258610    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"
Oct 30 19:20:33.258828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:33.258640    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9" event={"ID":"bec1b2e5-8104-403a-93b9-90bb0b3432f5","Type":"ContainerDied","Data":"f47cd1a114bda53379c748d626cfc0d22e556bcb725721b28beb69fd76a93cb3"}
Oct 30 19:20:33.258828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:33.258681    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f47cd1a114bda53379c748d626cfc0d22e556bcb725721b28beb69fd76a93cb3"
Oct 30 19:20:35.142147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.142111    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"]
Oct 30 19:20:35.142749 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.142727    6777 cpu_manager.go:414] "containerMap: removing container" podUID="bec1b2e5-8104-403a-93b9-90bb0b3432f5" containerName="revoke-keys"
Oct 30 19:20:35.142794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.142752    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec1b2e5-8104-403a-93b9-90bb0b3432f5" containerName="revoke-keys"
Oct 30 19:20:35.142934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.142906    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec1b2e5-8104-403a-93b9-90bb0b3432f5" containerName="revoke-keys"
Oct 30 19:20:35.146556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.146537    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.152754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.152729    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-39b867-openshift-default-dockercfg-8z9sp\""
Oct 30 19:20:35.164613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.164584    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"]
Oct 30 19:20:35.227327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.227289    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.227513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.227345    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.227513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.227456    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371c1da-8673-4248-880f-f99c6712a1e3-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.227513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.227506    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.227676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.227567    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.227676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.227623    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fzpk\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-kube-api-access-7fzpk\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.227885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.227853    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.227973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.227927    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371c1da-8673-4248-880f-f99c6712a1e3-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.228033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.228014    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.329559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.329513    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.329559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.329564    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.329802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.329588    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.329802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.329630    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371c1da-8673-4248-880f-f99c6712a1e3-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.329802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.329653    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.329802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.329709    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.329802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.329742    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fzpk\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-kube-api-access-7fzpk\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.330021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.329861    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.330021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.329895    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371c1da-8673-4248-880f-f99c6712a1e3-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.330195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.330170    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.330268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.330242    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.330322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.330280    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.330492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.330468    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371c1da-8673-4248-880f-f99c6712a1e3-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.330553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.330522    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.332071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.332037    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.332362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.332344    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371c1da-8673-4248-880f-f99c6712a1e3-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.343699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.343670    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.344045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.344024    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fzpk\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-kube-api-access-7fzpk\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.461345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.461256    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:35.751898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.751868    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"]
Oct 30 19:20:35.753355 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:20:35.753328    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1371c1da_8673_4248_880f_f99c6712a1e3.slice/crio-2bac3315d6657fd8aafe43d2aef190461e2d8b60567f00f27cfa4202c8545b12 WatchSource:0}: Error finding container 2bac3315d6657fd8aafe43d2aef190461e2d8b60567f00f27cfa4202c8545b12: Status 404 returned error can't find the container with id 2bac3315d6657fd8aafe43d2aef190461e2d8b60567f00f27cfa4202c8545b12
Oct 30 19:20:35.755871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.755837    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:20:35.755979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.755908    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:20:35.755979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:35.755939    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:20:36.278172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:36.278107    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" event={"ID":"1371c1da-8673-4248-880f-f99c6712a1e3","Type":"ContainerStarted","Data":"7f22391eefe354f45b1f91e52a6e1980fc19e6c0b535c7ca650894621a0ee868"}
Oct 30 19:20:36.278593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:36.278218    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" event={"ID":"1371c1da-8673-4248-880f-f99c6712a1e3","Type":"ContainerStarted","Data":"2bac3315d6657fd8aafe43d2aef190461e2d8b60567f00f27cfa4202c8545b12"}
Oct 30 19:20:36.305747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:36.305696    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podStartSLOduration=1.305677854 podStartE2EDuration="1.305677854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:36.303733274 +0000 UTC m=+818.300648661" watchObservedRunningTime="2026-10-30 19:20:36.305677854 +0000 UTC m=+818.302593236"
Oct 30 19:20:36.461854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:36.461820    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:36.463695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:36.463667    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:36.463832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:36.463715    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:20:36.505987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:36.505945    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:36.519774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:36.519737    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:37.295818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:37.295787    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:20:37.474645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:37.474603    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:37.474826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:37.474674    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:20:38.463678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:38.463636    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:38.464513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:38.464459    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:20:39.467085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:39.465862    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:39.467085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:39.465938    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:20:39.980084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:39.978814    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:20:40.007260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:40.007211    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:20:40.465560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:40.465511    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:40.465881 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:40.465850    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:20:41.471540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:41.471500    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:41.472174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:41.472139    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:20:42.462508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:42.462449    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:42.462718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:42.462525    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:20:43.648861 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:20:43.648818    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:20:44.351041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.351005    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:44.351508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.351477    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:44.352324 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.352291    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:20:44.701755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.701667    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"]
Oct 30 19:20:44.705898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.705874    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:44.716728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.716688    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"]
Oct 30 19:20:44.717310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.717284    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:44.717462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.717359    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:20:44.749454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.749421    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e12cbcb1-9zp5n\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:44.749647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.749559    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4fdn\" (UniqueName: \"kubernetes.io/projected/8c00d144-4aa7-43a2-9e99-21567b37d02e-kube-api-access-h4fdn\") pod \"maas-api-revoke-sub-e12cbcb1-9zp5n\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:44.749647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.749627    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e12cbcb1-9zp5n\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:44.851648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.851607    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e12cbcb1-9zp5n\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:44.852115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.852084    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e12cbcb1-9zp5n\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:44.852278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.852199    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4fdn\" (UniqueName: \"kubernetes.io/projected/8c00d144-4aa7-43a2-9e99-21567b37d02e-kube-api-access-h4fdn\") pod \"maas-api-revoke-sub-e12cbcb1-9zp5n\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:44.853265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.853163    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e12cbcb1-9zp5n\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:44.854769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.854743    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e12cbcb1-9zp5n\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:44.865481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:44.865455    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4fdn\" (UniqueName: \"kubernetes.io/projected/8c00d144-4aa7-43a2-9e99-21567b37d02e-kube-api-access-h4fdn\") pod \"maas-api-revoke-sub-e12cbcb1-9zp5n\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:45.022410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.022313    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:45.300543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.300518    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"]
Oct 30 19:20:45.302267 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:20:45.302240    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c00d144_4aa7_43a2_9e99_21567b37d02e.slice/crio-35802443d22e73782c90bf7ea8ff1871e9bdd3fa5fbfd42aa6f57862e49d1831 WatchSource:0}: Error finding container 35802443d22e73782c90bf7ea8ff1871e9bdd3fa5fbfd42aa6f57862e49d1831: Status 404 returned error can't find the container with id 35802443d22e73782c90bf7ea8ff1871e9bdd3fa5fbfd42aa6f57862e49d1831
Oct 30 19:20:45.416640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.416596    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n" event={"ID":"8c00d144-4aa7-43a2-9e99-21567b37d02e","Type":"ContainerStarted","Data":"35802443d22e73782c90bf7ea8ff1871e9bdd3fa5fbfd42aa6f57862e49d1831"}
Oct 30 19:20:45.802255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.802213    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"]
Oct 30 19:20:45.806308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.806283    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:45.815513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.815488    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"]
Oct 30 19:20:45.867685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.867648    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a12eb4-k4ndn\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:45.867854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.867828    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6q7\" (UniqueName: \"kubernetes.io/projected/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-kube-api-access-sq6q7\") pod \"maas-api-revoke-sub-91a12eb4-k4ndn\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:45.868025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.868005    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a12eb4-k4ndn\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:45.969356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.969317    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a12eb4-k4ndn\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:45.969562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.969370    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a12eb4-k4ndn\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:45.969562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.969507    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6q7\" (UniqueName: \"kubernetes.io/projected/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-kube-api-access-sq6q7\") pod \"maas-api-revoke-sub-91a12eb4-k4ndn\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:45.970026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.969995    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a12eb4-k4ndn\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:45.971850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.971829    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a12eb4-k4ndn\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:45.981921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:45.981892    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6q7\" (UniqueName: \"kubernetes.io/projected/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-kube-api-access-sq6q7\") pod \"maas-api-revoke-sub-91a12eb4-k4ndn\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:46.121511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:46.121421    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:46.409096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:46.409053    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"]
Oct 30 19:20:46.410722 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:20:46.410694    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f7c9f5_bc4d_4397_8136_c595a93d6d7d.slice/crio-a61323fe9f0210d870bfcd89ed38eded89aee6fdbcf978bc556f765c3ee6999e WatchSource:0}: Error finding container a61323fe9f0210d870bfcd89ed38eded89aee6fdbcf978bc556f765c3ee6999e: Status 404 returned error can't find the container with id a61323fe9f0210d870bfcd89ed38eded89aee6fdbcf978bc556f765c3ee6999e
Oct 30 19:20:46.423189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:46.423160    6777 generic.go:356] "Generic (PLEG): container finished" podID="8c00d144-4aa7-43a2-9e99-21567b37d02e" containerID="7302c8d4147e3f64d49075aa49fb4af76fafc749c256efec7f532f6dccf8d2af" exitCode=0
Oct 30 19:20:46.423320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:46.423230    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n" event={"ID":"8c00d144-4aa7-43a2-9e99-21567b37d02e","Type":"ContainerDied","Data":"7302c8d4147e3f64d49075aa49fb4af76fafc749c256efec7f532f6dccf8d2af"}
Oct 30 19:20:46.424561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:46.424539    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn" event={"ID":"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d","Type":"ContainerStarted","Data":"a61323fe9f0210d870bfcd89ed38eded89aee6fdbcf978bc556f765c3ee6999e"}
Oct 30 19:20:47.443857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.443816    6777 generic.go:356] "Generic (PLEG): container finished" podID="f9f7c9f5-bc4d-4397-8136-c595a93d6d7d" containerID="64e40ca3173f5205af2908f5c52f3be1a62fd2cc4f2c4f28ebedc71c1a66eb16" exitCode=0
Oct 30 19:20:47.444349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.443905    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn" event={"ID":"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d","Type":"ContainerDied","Data":"64e40ca3173f5205af2908f5c52f3be1a62fd2cc4f2c4f28ebedc71c1a66eb16"}
Oct 30 19:20:47.607420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.607390    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:47.695822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.695784    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c00d144-4aa7-43a2-9e99-21567b37d02e-kube-api-access-h4fdn\" (UniqueName: \"kubernetes.io/projected/8c00d144-4aa7-43a2-9e99-21567b37d02e-kube-api-access-h4fdn\") pod \"8c00d144-4aa7-43a2-9e99-21567b37d02e\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") "
Oct 30 19:20:47.696012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.695905    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-openshift-ca\") pod \"8c00d144-4aa7-43a2-9e99-21567b37d02e\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") "
Oct 30 19:20:47.696117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.696028    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-xks-ca\") pod \"8c00d144-4aa7-43a2-9e99-21567b37d02e\" (UID: \"8c00d144-4aa7-43a2-9e99-21567b37d02e\") "
Oct 30 19:20:47.696463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.696432    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-openshift-ca" pod "8c00d144-4aa7-43a2-9e99-21567b37d02e" (UID: "8c00d144-4aa7-43a2-9e99-21567b37d02e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:47.696602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.696581    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:47.699169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.699137    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c00d144-4aa7-43a2-9e99-21567b37d02e-kube-api-access-h4fdn" pod "8c00d144-4aa7-43a2-9e99-21567b37d02e" (UID: "8c00d144-4aa7-43a2-9e99-21567b37d02e"). InnerVolumeSpecName "kube-api-access-h4fdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:47.699268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.699244    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-xks-ca" pod "8c00d144-4aa7-43a2-9e99-21567b37d02e" (UID: "8c00d144-4aa7-43a2-9e99-21567b37d02e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:47.797621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.797575    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c00d144-4aa7-43a2-9e99-21567b37d02e-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:47.797621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:47.797613    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4fdn\" (UniqueName: \"kubernetes.io/projected/8c00d144-4aa7-43a2-9e99-21567b37d02e-kube-api-access-h4fdn\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:48.466468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.463137    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"
Oct 30 19:20:48.466468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.463966    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n" event={"ID":"8c00d144-4aa7-43a2-9e99-21567b37d02e","Type":"ContainerDied","Data":"35802443d22e73782c90bf7ea8ff1871e9bdd3fa5fbfd42aa6f57862e49d1831"}
Oct 30 19:20:48.466468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.464017    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35802443d22e73782c90bf7ea8ff1871e9bdd3fa5fbfd42aa6f57862e49d1831"
Oct 30 19:20:48.630678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.630648    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:48.708264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.708230    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-openshift-ca\") pod \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") "
Oct 30 19:20:48.708445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.708363    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-xks-ca\") pod \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") "
Oct 30 19:20:48.708445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.708384    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-kube-api-access-sq6q7\" (UniqueName: \"kubernetes.io/projected/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-kube-api-access-sq6q7\") pod \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\" (UID: \"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d\") "
Oct 30 19:20:48.708684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.708657    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-openshift-ca" pod "f9f7c9f5-bc4d-4397-8136-c595a93d6d7d" (UID: "f9f7c9f5-bc4d-4397-8136-c595a93d6d7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:48.708856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.708841    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:48.710436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.710411    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-xks-ca" pod "f9f7c9f5-bc4d-4397-8136-c595a93d6d7d" (UID: "f9f7c9f5-bc4d-4397-8136-c595a93d6d7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:48.710436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.710429    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-kube-api-access-sq6q7" pod "f9f7c9f5-bc4d-4397-8136-c595a93d6d7d" (UID: "f9f7c9f5-bc4d-4397-8136-c595a93d6d7d"). InnerVolumeSpecName "kube-api-access-sq6q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:48.809659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.809624    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:48.809659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:48.809655    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq6q7\" (UniqueName: \"kubernetes.io/projected/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d-kube-api-access-sq6q7\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.469928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.469893    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn" event={"ID":"f9f7c9f5-bc4d-4397-8136-c595a93d6d7d","Type":"ContainerDied","Data":"a61323fe9f0210d870bfcd89ed38eded89aee6fdbcf978bc556f765c3ee6999e"}
Oct 30 19:20:49.470417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.469954    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a61323fe9f0210d870bfcd89ed38eded89aee6fdbcf978bc556f765c3ee6999e"
Oct 30 19:20:49.470417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.469902    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"
Oct 30 19:20:49.934212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.934174    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"]
Oct 30 19:20:49.934703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.934689    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8c00d144-4aa7-43a2-9e99-21567b37d02e" containerName="revoke-keys"
Oct 30 19:20:49.934746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.934705    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c00d144-4aa7-43a2-9e99-21567b37d02e" containerName="revoke-keys"
Oct 30 19:20:49.934746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.934733    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f9f7c9f5-bc4d-4397-8136-c595a93d6d7d" containerName="revoke-keys"
Oct 30 19:20:49.934746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.934738    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f7c9f5-bc4d-4397-8136-c595a93d6d7d" containerName="revoke-keys"
Oct 30 19:20:49.934841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.934827    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f7c9f5-bc4d-4397-8136-c595a93d6d7d" containerName="revoke-keys"
Oct 30 19:20:49.934873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.934841    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c00d144-4aa7-43a2-9e99-21567b37d02e" containerName="revoke-keys"
Oct 30 19:20:49.938300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.938285    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:49.950129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.950101    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2d5d53-openshift-default-dockercfg-mbh26\""
Oct 30 19:20:49.953671 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:49.953643    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"]
Oct 30 19:20:50.030550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.030496    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.030748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.030561    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.030748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.030647    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ca458c-b3a8-4c14-93b5-4da753fbc136-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.030748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.030706    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.030880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.030826    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.030880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.030859    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.030880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.030877    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.031132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.031105    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkt4\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-kube-api-access-zbkt4\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.031269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.031187    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.065297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.065260    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:20:50.134814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.134769    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.135008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.134835    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.135008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.134904    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkt4\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-kube-api-access-zbkt4\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.135008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.134990    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.135216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.135139    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.135216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.135176    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.135216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.135209    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ca458c-b3a8-4c14-93b5-4da753fbc136-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.135360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.135296    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.135409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.135398    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.138308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.138270    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.138905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.138875    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ca458c-b3a8-4c14-93b5-4da753fbc136-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.139246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.139221    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.140271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.140242    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.140663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.140639    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.146658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.146625    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.150873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.150851    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.152706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.152676    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.152840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.152818    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkt4\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-kube-api-access-zbkt4\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.252314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.252209    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:50.537954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.537927    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"]
Oct 30 19:20:50.543383 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:20:50.542186    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ca458c_b3a8_4c14_93b5_4da753fbc136.slice/crio-8b3f806141d369c0dcd4bc7be97f2272893a2514c89b20417e19f6fed5fb7a71 WatchSource:0}: Error finding container 8b3f806141d369c0dcd4bc7be97f2272893a2514c89b20417e19f6fed5fb7a71: Status 404 returned error can't find the container with id 8b3f806141d369c0dcd4bc7be97f2272893a2514c89b20417e19f6fed5fb7a71
Oct 30 19:20:50.545366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.545321    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:20:50.545510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.545420    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:20:50.545510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:50.545464    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:20:51.543017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:51.542968    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" event={"ID":"96ca458c-b3a8-4c14-93b5-4da753fbc136","Type":"ContainerStarted","Data":"cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9"}
Oct 30 19:20:51.544886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:51.544841    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" event={"ID":"96ca458c-b3a8-4c14-93b5-4da753fbc136","Type":"ContainerStarted","Data":"8b3f806141d369c0dcd4bc7be97f2272893a2514c89b20417e19f6fed5fb7a71"}
Oct 30 19:20:51.610100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:51.595087    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podStartSLOduration=2.595040273 podStartE2EDuration="2.595040273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:51.58127851 +0000 UTC m=+833.578193910" watchObservedRunningTime="2026-10-30 19:20:51.595040273 +0000 UTC m=+833.591955655"
Oct 30 19:20:52.269082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:52.266093    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:52.269082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:52.266173    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:20:52.269082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:52.266512    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:53.269083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:53.264501    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:53.269083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:53.264569    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:20:54.256583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:54.256544    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:54.256891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:54.256858    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:20:55.256095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:55.255407    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:55.256095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:55.255474    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:20:56.262083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:56.259477    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:56.262083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:56.259541    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:20:57.039894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.039860    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"]
Oct 30 19:20:57.045212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.045188    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.052327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.052274    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:57.052568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.052548    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:20:57.052693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.052570    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"]
Oct 30 19:20:57.199964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.199931    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bee26fe-wwx94\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.200160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.200046    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtrs\" (UniqueName: \"kubernetes.io/projected/0cab960a-94b0-4964-b3b4-8a61c4a37a49-kube-api-access-mxtrs\") pod \"maas-api-revoke-sub-6bee26fe-wwx94\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.200257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.200241    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bee26fe-wwx94\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.253459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.253428    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:57.253639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.253485    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:20:57.301207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.301131    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bee26fe-wwx94\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.301663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.301227    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtrs\" (UniqueName: \"kubernetes.io/projected/0cab960a-94b0-4964-b3b4-8a61c4a37a49-kube-api-access-mxtrs\") pod \"maas-api-revoke-sub-6bee26fe-wwx94\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.301663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.301398    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bee26fe-wwx94\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.302351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.302326    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bee26fe-wwx94\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.303567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.303534    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bee26fe-wwx94\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.314452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.314424    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtrs\" (UniqueName: \"kubernetes.io/projected/0cab960a-94b0-4964-b3b4-8a61c4a37a49-kube-api-access-mxtrs\") pod \"maas-api-revoke-sub-6bee26fe-wwx94\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.372532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.372236    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:57.678583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:57.678541    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"]
Oct 30 19:20:57.679703 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:20:57.679671    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cab960a_94b0_4964_b3b4_8a61c4a37a49.slice/crio-92fa502725d5df1f4fe8b655fd11c2ac87c7b6e6c7e27d175bc0a22e8365a520 WatchSource:0}: Error finding container 92fa502725d5df1f4fe8b655fd11c2ac87c7b6e6c7e27d175bc0a22e8365a520: Status 404 returned error can't find the container with id 92fa502725d5df1f4fe8b655fd11c2ac87c7b6e6c7e27d175bc0a22e8365a520
Oct 30 19:20:58.257244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.257173    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:58.257469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.257448    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:58.258076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.258041    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:20:58.613558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.613526    6777 generic.go:356] "Generic (PLEG): container finished" podID="0cab960a-94b0-4964-b3b4-8a61c4a37a49" containerID="7e008d6f66251eb2156095ad618959b207c74e17859394e5d3e257c71956f550" exitCode=0
Oct 30 19:20:58.613882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.613611    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94" event={"ID":"0cab960a-94b0-4964-b3b4-8a61c4a37a49","Type":"ContainerDied","Data":"7e008d6f66251eb2156095ad618959b207c74e17859394e5d3e257c71956f550"}
Oct 30 19:20:58.613882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.613666    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94" event={"ID":"0cab960a-94b0-4964-b3b4-8a61c4a37a49","Type":"ContainerStarted","Data":"92fa502725d5df1f4fe8b655fd11c2ac87c7b6e6c7e27d175bc0a22e8365a520"}
Oct 30 19:20:58.808095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.807998    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"]
Oct 30 19:20:58.811867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.811844    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:58.823500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.821811    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"]
Oct 30 19:20:58.930632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.930597    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnff\" (UniqueName: \"kubernetes.io/projected/98f2acce-36f7-49af-add9-5d36c81aafa0-kube-api-access-lbnff\") pod \"maas-api-revoke-sub-889ffa11-p4dmz\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:58.930807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.930730    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-889ffa11-p4dmz\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:58.930852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:58.930822    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-889ffa11-p4dmz\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:59.032402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.032372    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-889ffa11-p4dmz\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:59.032581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.032451    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnff\" (UniqueName: \"kubernetes.io/projected/98f2acce-36f7-49af-add9-5d36c81aafa0-kube-api-access-lbnff\") pod \"maas-api-revoke-sub-889ffa11-p4dmz\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:59.032581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.032557    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-889ffa11-p4dmz\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:59.033262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.033237    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-889ffa11-p4dmz\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:59.034800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.034780    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-889ffa11-p4dmz\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:59.045962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.045941    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnff\" (UniqueName: \"kubernetes.io/projected/98f2acce-36f7-49af-add9-5d36c81aafa0-kube-api-access-lbnff\") pod \"maas-api-revoke-sub-889ffa11-p4dmz\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:59.128453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.128387    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:20:59.417090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.417033    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"]
Oct 30 19:20:59.419499 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:20:59.419472    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f2acce_36f7_49af_add9_5d36c81aafa0.slice/crio-acbef9ca12678d8bf41273fd7f415f780406cb5033b44801490ed1c64dd95495 WatchSource:0}: Error finding container acbef9ca12678d8bf41273fd7f415f780406cb5033b44801490ed1c64dd95495: Status 404 returned error can't find the container with id acbef9ca12678d8bf41273fd7f415f780406cb5033b44801490ed1c64dd95495
Oct 30 19:20:59.620250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.620213    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz" event={"ID":"98f2acce-36f7-49af-add9-5d36c81aafa0","Type":"ContainerStarted","Data":"acbef9ca12678d8bf41273fd7f415f780406cb5033b44801490ed1c64dd95495"}
Oct 30 19:20:59.785332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.785305    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:20:59.944149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.944050    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-xks-ca\") pod \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") "
Oct 30 19:20:59.944149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.944122    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cab960a-94b0-4964-b3b4-8a61c4a37a49-kube-api-access-mxtrs\" (UniqueName: \"kubernetes.io/projected/0cab960a-94b0-4964-b3b4-8a61c4a37a49-kube-api-access-mxtrs\") pod \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") "
Oct 30 19:20:59.944352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.944225    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-openshift-ca\") pod \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\" (UID: \"0cab960a-94b0-4964-b3b4-8a61c4a37a49\") "
Oct 30 19:20:59.944560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.944530    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-openshift-ca" pod "0cab960a-94b0-4964-b3b4-8a61c4a37a49" (UID: "0cab960a-94b0-4964-b3b4-8a61c4a37a49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:59.944764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.944750    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:59.946148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.946130    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-xks-ca" pod "0cab960a-94b0-4964-b3b4-8a61c4a37a49" (UID: "0cab960a-94b0-4964-b3b4-8a61c4a37a49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:59.946252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:20:59.946233    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cab960a-94b0-4964-b3b4-8a61c4a37a49-kube-api-access-mxtrs" pod "0cab960a-94b0-4964-b3b4-8a61c4a37a49" (UID: "0cab960a-94b0-4964-b3b4-8a61c4a37a49"). InnerVolumeSpecName "kube-api-access-mxtrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:00.046127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.046099    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cab960a-94b0-4964-b3b4-8a61c4a37a49-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:00.046127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.046122    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtrs\" (UniqueName: \"kubernetes.io/projected/0cab960a-94b0-4964-b3b4-8a61c4a37a49-kube-api-access-mxtrs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:00.627526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.627491    6777 generic.go:356] "Generic (PLEG): container finished" podID="98f2acce-36f7-49af-add9-5d36c81aafa0" containerID="455dfb379cbe260b91c41cd84efadad900de33923a70d712d9d7d3e95703971d" exitCode=0
Oct 30 19:21:00.627949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.627581    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz" event={"ID":"98f2acce-36f7-49af-add9-5d36c81aafa0","Type":"ContainerDied","Data":"455dfb379cbe260b91c41cd84efadad900de33923a70d712d9d7d3e95703971d"}
Oct 30 19:21:00.628888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.628870    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"
Oct 30 19:21:00.628991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.628903    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94" event={"ID":"0cab960a-94b0-4964-b3b4-8a61c4a37a49","Type":"ContainerDied","Data":"92fa502725d5df1f4fe8b655fd11c2ac87c7b6e6c7e27d175bc0a22e8365a520"}
Oct 30 19:21:00.628991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.628935    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92fa502725d5df1f4fe8b655fd11c2ac87c7b6e6c7e27d175bc0a22e8365a520"
Oct 30 19:21:00.897042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.896954    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"]
Oct 30 19:21:00.897520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.897505    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0cab960a-94b0-4964-b3b4-8a61c4a37a49" containerName="revoke-keys"
Oct 30 19:21:00.897559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.897521    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cab960a-94b0-4964-b3b4-8a61c4a37a49" containerName="revoke-keys"
Oct 30 19:21:00.897634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.897625    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cab960a-94b0-4964-b3b4-8a61c4a37a49" containerName="revoke-keys"
Oct 30 19:21:00.900869 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.900855    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:00.909722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:00.909699    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"]
Oct 30 19:21:01.059284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.059254    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9hp\" (UniqueName: \"kubernetes.io/projected/ead81709-4ac0-4d00-9166-92e267eb685c-kube-api-access-rq9hp\") pod \"maas-api-revoke-sub-9ec55f3b-nw6j5\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.059452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.059427    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ec55f3b-nw6j5\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.059495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.059454    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ec55f3b-nw6j5\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.160067 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.159966    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ec55f3b-nw6j5\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.160067 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.160004    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ec55f3b-nw6j5\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.160067 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.160047    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9hp\" (UniqueName: \"kubernetes.io/projected/ead81709-4ac0-4d00-9166-92e267eb685c-kube-api-access-rq9hp\") pod \"maas-api-revoke-sub-9ec55f3b-nw6j5\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.160744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.160689    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ec55f3b-nw6j5\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.162427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.162403    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ec55f3b-nw6j5\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.173362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.173335    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9hp\" (UniqueName: \"kubernetes.io/projected/ead81709-4ac0-4d00-9166-92e267eb685c-kube-api-access-rq9hp\") pod \"maas-api-revoke-sub-9ec55f3b-nw6j5\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.217200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.217171    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:01.506676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.506646    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"]
Oct 30 19:21:01.508045 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:01.508016    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead81709_4ac0_4d00_9166_92e267eb685c.slice/crio-5d34ddec877ccd8511078d8cecc1cabffb01041e47b1667a43462ae4c6e3331a WatchSource:0}: Error finding container 5d34ddec877ccd8511078d8cecc1cabffb01041e47b1667a43462ae4c6e3331a: Status 404 returned error can't find the container with id 5d34ddec877ccd8511078d8cecc1cabffb01041e47b1667a43462ae4c6e3331a
Oct 30 19:21:01.635875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.635841    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5" event={"ID":"ead81709-4ac0-4d00-9166-92e267eb685c","Type":"ContainerStarted","Data":"5d34ddec877ccd8511078d8cecc1cabffb01041e47b1667a43462ae4c6e3331a"}
Oct 30 19:21:01.785313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.785281    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:21:01.869874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.869840    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-openshift-ca\") pod \"98f2acce-36f7-49af-add9-5d36c81aafa0\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") "
Oct 30 19:21:01.870248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.870224    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f2acce-36f7-49af-add9-5d36c81aafa0-kube-api-access-lbnff\" (UniqueName: \"kubernetes.io/projected/98f2acce-36f7-49af-add9-5d36c81aafa0-kube-api-access-lbnff\") pod \"98f2acce-36f7-49af-add9-5d36c81aafa0\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") "
Oct 30 19:21:01.870368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.870315    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-openshift-ca" pod "98f2acce-36f7-49af-add9-5d36c81aafa0" (UID: "98f2acce-36f7-49af-add9-5d36c81aafa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:01.870622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.870601    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-xks-ca\") pod \"98f2acce-36f7-49af-add9-5d36c81aafa0\" (UID: \"98f2acce-36f7-49af-add9-5d36c81aafa0\") "
Oct 30 19:21:01.871615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.871589    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.876947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.876918    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-xks-ca" pod "98f2acce-36f7-49af-add9-5d36c81aafa0" (UID: "98f2acce-36f7-49af-add9-5d36c81aafa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:01.877205 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.877183    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f2acce-36f7-49af-add9-5d36c81aafa0-kube-api-access-lbnff" pod "98f2acce-36f7-49af-add9-5d36c81aafa0" (UID: "98f2acce-36f7-49af-add9-5d36c81aafa0"). InnerVolumeSpecName "kube-api-access-lbnff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:01.973279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.973251    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnff\" (UniqueName: \"kubernetes.io/projected/98f2acce-36f7-49af-add9-5d36c81aafa0-kube-api-access-lbnff\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.973279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:01.973281    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f2acce-36f7-49af-add9-5d36c81aafa0-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.695481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:02.692970    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz" event={"ID":"98f2acce-36f7-49af-add9-5d36c81aafa0","Type":"ContainerDied","Data":"acbef9ca12678d8bf41273fd7f415f780406cb5033b44801490ed1c64dd95495"}
Oct 30 19:21:02.695481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:02.693040    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acbef9ca12678d8bf41273fd7f415f780406cb5033b44801490ed1c64dd95495"
Oct 30 19:21:02.695481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:02.693143    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"
Oct 30 19:21:02.706414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:02.706335    6777 generic.go:356] "Generic (PLEG): container finished" podID="ead81709-4ac0-4d00-9166-92e267eb685c" containerID="f46093d6fc3b374b9669e6caa5736b86bb963c9ce8003f23444c44febc3e1a57" exitCode=0
Oct 30 19:21:02.706639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:02.706616    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5" event={"ID":"ead81709-4ac0-4d00-9166-92e267eb685c","Type":"ContainerDied","Data":"f46093d6fc3b374b9669e6caa5736b86bb963c9ce8003f23444c44febc3e1a57"}
Oct 30 19:21:03.858123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:03.858093    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:04.005348 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.005262    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-xks-ca\") pod \"ead81709-4ac0-4d00-9166-92e267eb685c\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") "
Oct 30 19:21:04.005506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.005406    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-openshift-ca\") pod \"ead81709-4ac0-4d00-9166-92e267eb685c\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") "
Oct 30 19:21:04.005506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.005486    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead81709-4ac0-4d00-9166-92e267eb685c-kube-api-access-rq9hp\" (UniqueName: \"kubernetes.io/projected/ead81709-4ac0-4d00-9166-92e267eb685c-kube-api-access-rq9hp\") pod \"ead81709-4ac0-4d00-9166-92e267eb685c\" (UID: \"ead81709-4ac0-4d00-9166-92e267eb685c\") "
Oct 30 19:21:04.005750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.005716    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-openshift-ca" pod "ead81709-4ac0-4d00-9166-92e267eb685c" (UID: "ead81709-4ac0-4d00-9166-92e267eb685c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:04.006701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.005923    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:04.007474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.007452    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-xks-ca" pod "ead81709-4ac0-4d00-9166-92e267eb685c" (UID: "ead81709-4ac0-4d00-9166-92e267eb685c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:04.007892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.007871    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead81709-4ac0-4d00-9166-92e267eb685c-kube-api-access-rq9hp" pod "ead81709-4ac0-4d00-9166-92e267eb685c" (UID: "ead81709-4ac0-4d00-9166-92e267eb685c"). InnerVolumeSpecName "kube-api-access-rq9hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:04.106890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.106859    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead81709-4ac0-4d00-9166-92e267eb685c-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:04.106890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.106883    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq9hp\" (UniqueName: \"kubernetes.io/projected/ead81709-4ac0-4d00-9166-92e267eb685c-kube-api-access-rq9hp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:04.722605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.722573    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"
Oct 30 19:21:04.722919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.722604    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5" event={"ID":"ead81709-4ac0-4d00-9166-92e267eb685c","Type":"ContainerDied","Data":"5d34ddec877ccd8511078d8cecc1cabffb01041e47b1667a43462ae4c6e3331a"}
Oct 30 19:21:04.722919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.722829    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d34ddec877ccd8511078d8cecc1cabffb01041e47b1667a43462ae4c6e3331a"
Oct 30 19:21:04.801962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.801928    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"]
Oct 30 19:21:04.802756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.802733    6777 cpu_manager.go:414] "containerMap: removing container" podUID="98f2acce-36f7-49af-add9-5d36c81aafa0" containerName="revoke-keys"
Oct 30 19:21:04.802857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.802759    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f2acce-36f7-49af-add9-5d36c81aafa0" containerName="revoke-keys"
Oct 30 19:21:04.802857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.802816    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ead81709-4ac0-4d00-9166-92e267eb685c" containerName="revoke-keys"
Oct 30 19:21:04.802857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.802824    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead81709-4ac0-4d00-9166-92e267eb685c" containerName="revoke-keys"
Oct 30 19:21:04.802972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.802961    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f2acce-36f7-49af-add9-5d36c81aafa0" containerName="revoke-keys"
Oct 30 19:21:04.803006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.802981    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead81709-4ac0-4d00-9166-92e267eb685c" containerName="revoke-keys"
Oct 30 19:21:04.808196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.807268    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:04.820742 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.820484    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"]
Oct 30 19:21:04.919895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.919863    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:04.919895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.919893    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:04.920474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.920012    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:04.920474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.920129    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:04.920474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.920188    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:04.920474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.920273    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:04.920474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.920410    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:04.920474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.920439    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:04.920474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:04.920465    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwfqj\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-kube-api-access-nwfqj\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.021829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.021753    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.021829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.021807    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.021997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.021967    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022045    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022218    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022309 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022248    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022309 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022276    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwfqj\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-kube-api-access-nwfqj\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022323    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022374    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022403    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022566    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022713    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.022838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.022817    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.023189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.023171    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.024229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.024206    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.024326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.024262    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.035006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.034967    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.035199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.035176    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwfqj\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-kube-api-access-nwfqj\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.127164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.127133    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:05.456155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.456125    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"]
Oct 30 19:21:05.457279 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:05.457253    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c80e7ec_fe29_4139_a608_d10d0fb4901d.slice/crio-a1ef05303f93a3a2c471a6e167859b261a6a4e9173dd4b46c4a47a9096f99ca9 WatchSource:0}: Error finding container a1ef05303f93a3a2c471a6e167859b261a6a4e9173dd4b46c4a47a9096f99ca9: Status 404 returned error can't find the container with id a1ef05303f93a3a2c471a6e167859b261a6a4e9173dd4b46c4a47a9096f99ca9
Oct 30 19:21:05.459896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.459861    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:05.459998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.459977    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:05.460081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.460020    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:05.730968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.730877    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" event={"ID":"5c80e7ec-fe29-4139-a608-d10d0fb4901d","Type":"ContainerStarted","Data":"af2f36699131ef9ef25942df56ef8e2e3dd969fb36c522aeff0ace152104d833"}
Oct 30 19:21:05.730968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.730938    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" event={"ID":"5c80e7ec-fe29-4139-a608-d10d0fb4901d","Type":"ContainerStarted","Data":"a1ef05303f93a3a2c471a6e167859b261a6a4e9173dd4b46c4a47a9096f99ca9"}
Oct 30 19:21:05.755981 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:05.755929    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" podStartSLOduration=1.75591374 podStartE2EDuration="1.75591374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:05.754148045 +0000 UTC m=+847.751063428" watchObservedRunningTime="2026-10-30 19:21:05.75591374 +0000 UTC m=+847.752829122"
Oct 30 19:21:06.127978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:06.127949    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:06.129243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:06.129220    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:21:06.129316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:06.129266    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:21:07.127830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:07.127792    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:21:07.128047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:07.127855    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:21:08.128052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:08.128019    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:21:08.128473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:08.128110    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:21:09.128372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:09.128342    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:21:09.128778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:09.128394    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:21:10.127857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.127822    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:21:10.128033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.127887    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:21:10.345973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.345943    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"]
Oct 30 19:21:10.371809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.371780    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.378921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.378854    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:21:10.379220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.379174    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:10.379564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.379542    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"]
Oct 30 19:21:10.511809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.511767    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdxm\" (UniqueName: \"kubernetes.io/projected/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-kube-api-access-7wdxm\") pod \"maas-api-revoke-sub-ad1788af-sqthl\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.512005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.511864    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1788af-sqthl\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.512005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.511899    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1788af-sqthl\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.612679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.612639    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdxm\" (UniqueName: \"kubernetes.io/projected/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-kube-api-access-7wdxm\") pod \"maas-api-revoke-sub-ad1788af-sqthl\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.612826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.612751    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1788af-sqthl\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.612826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.612789    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1788af-sqthl\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.613577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.613551    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1788af-sqthl\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.615431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.615407    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1788af-sqthl\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.628094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.628048    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdxm\" (UniqueName: \"kubernetes.io/projected/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-kube-api-access-7wdxm\") pod \"maas-api-revoke-sub-ad1788af-sqthl\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.691852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.691770    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:10.996172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:10.996145    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"]
Oct 30 19:21:10.997787 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:10.997760    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34252ef3_3dd7_48c4_9bde_af9f27ffc3f7.slice/crio-42ed1ffec25dc8e5885946240c7dee0cf50bc336b651c93a5700ae257d424ea9 WatchSource:0}: Error finding container 42ed1ffec25dc8e5885946240c7dee0cf50bc336b651c93a5700ae257d424ea9: Status 404 returned error can't find the container with id 42ed1ffec25dc8e5885946240c7dee0cf50bc336b651c93a5700ae257d424ea9
Oct 30 19:21:11.127944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:11.127908    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:21:11.128135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:11.127962    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:21:11.806196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:11.806164    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl" event={"ID":"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7","Type":"ContainerStarted","Data":"b2574e8b7b21d74a9adbcf57c19ddbd2f6d03ce0096623344d25c96c506e49e5"}
Oct 30 19:21:11.806688 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:11.806212    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl" event={"ID":"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7","Type":"ContainerStarted","Data":"42ed1ffec25dc8e5885946240c7dee0cf50bc336b651c93a5700ae257d424ea9"}
Oct 30 19:21:11.828624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:11.828556    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl" podStartSLOduration=1.82854275 podStartE2EDuration="1.82854275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:11.826767503 +0000 UTC m=+853.823682888" watchObservedRunningTime="2026-10-30 19:21:11.82854275 +0000 UTC m=+853.825458188"
Oct 30 19:21:12.132204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:12.132126    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:12.132461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:12.132426    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:12.133251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:12.133235    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:21:12.814244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:12.814211    6777 generic.go:356] "Generic (PLEG): container finished" podID="34252ef3-3dd7-48c4-9bde-af9f27ffc3f7" containerID="b2574e8b7b21d74a9adbcf57c19ddbd2f6d03ce0096623344d25c96c506e49e5" exitCode=0
Oct 30 19:21:12.814651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:12.814289    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl" event={"ID":"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7","Type":"ContainerDied","Data":"b2574e8b7b21d74a9adbcf57c19ddbd2f6d03ce0096623344d25c96c506e49e5"}
Oct 30 19:21:13.418173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.418140    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"]
Oct 30 19:21:13.422136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.422120    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.430002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.429981    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"]
Oct 30 19:21:13.455008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.454978    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38f62e31-fldfj\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.455179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.455149    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrgk\" (UniqueName: \"kubernetes.io/projected/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-kube-api-access-gjrgk\") pod \"maas-api-revoke-sub-38f62e31-fldfj\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.455277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.455257    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38f62e31-fldfj\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.556995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.556959    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38f62e31-fldfj\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.557212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.557028    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrgk\" (UniqueName: \"kubernetes.io/projected/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-kube-api-access-gjrgk\") pod \"maas-api-revoke-sub-38f62e31-fldfj\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.557212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.557110    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38f62e31-fldfj\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.557781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.557756    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38f62e31-fldfj\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.559584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.559558    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38f62e31-fldfj\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.571826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.571801    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrgk\" (UniqueName: \"kubernetes.io/projected/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-kube-api-access-gjrgk\") pod \"maas-api-revoke-sub-38f62e31-fldfj\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.639860 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:21:13.639822    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:21:13.739670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.739582    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:13.997266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:13.997211    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:14.014641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.014615    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"]
Oct 30 19:21:14.014825 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:14.014801    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fd068c_11fc_4e31_b07b_c7653cbeb71c.slice/crio-7beeeb00cc24e0a46ffbf13516e5ce380f868acf0be9be09bdd083be588a5aa3 WatchSource:0}: Error finding container 7beeeb00cc24e0a46ffbf13516e5ce380f868acf0be9be09bdd083be588a5aa3: Status 404 returned error can't find the container with id 7beeeb00cc24e0a46ffbf13516e5ce380f868acf0be9be09bdd083be588a5aa3
Oct 30 19:21:14.074151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.074124    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-kube-api-access-7wdxm\" (UniqueName: \"kubernetes.io/projected/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-kube-api-access-7wdxm\") pod \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") "
Oct 30 19:21:14.074279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.074166    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-xks-ca\") pod \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") "
Oct 30 19:21:14.074345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.074333    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-openshift-ca\") pod \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\" (UID: \"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7\") "
Oct 30 19:21:14.074727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.074696    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-openshift-ca" pod "34252ef3-3dd7-48c4-9bde-af9f27ffc3f7" (UID: "34252ef3-3dd7-48c4-9bde-af9f27ffc3f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:14.076360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.076337    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-kube-api-access-7wdxm" pod "34252ef3-3dd7-48c4-9bde-af9f27ffc3f7" (UID: "34252ef3-3dd7-48c4-9bde-af9f27ffc3f7"). InnerVolumeSpecName "kube-api-access-7wdxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:14.076654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.076627    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-xks-ca" pod "34252ef3-3dd7-48c4-9bde-af9f27ffc3f7" (UID: "34252ef3-3dd7-48c4-9bde-af9f27ffc3f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:14.175497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.175462    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:14.175497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.175493    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wdxm\" (UniqueName: \"kubernetes.io/projected/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-kube-api-access-7wdxm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:14.175497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.175506    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:14.832914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.832797    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj" event={"ID":"c9fd068c-11fc-4e31-b07b-c7653cbeb71c","Type":"ContainerStarted","Data":"8aa953bfe0f9c425166ec64e0a2aa0316b038f77f9b0bc06a48b4a5a05446bd8"}
Oct 30 19:21:14.832914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.832905    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj" event={"ID":"c9fd068c-11fc-4e31-b07b-c7653cbeb71c","Type":"ContainerStarted","Data":"7beeeb00cc24e0a46ffbf13516e5ce380f868acf0be9be09bdd083be588a5aa3"}
Oct 30 19:21:14.834351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.834324    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"
Oct 30 19:21:14.834502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.834329    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl" event={"ID":"34252ef3-3dd7-48c4-9bde-af9f27ffc3f7","Type":"ContainerDied","Data":"42ed1ffec25dc8e5885946240c7dee0cf50bc336b651c93a5700ae257d424ea9"}
Oct 30 19:21:14.834573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.834518    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ed1ffec25dc8e5885946240c7dee0cf50bc336b651c93a5700ae257d424ea9"
Oct 30 19:21:14.852449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:14.852403    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj" podStartSLOduration=1.852390979 podStartE2EDuration="1.852390979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:14.85139053 +0000 UTC m=+856.848305945" watchObservedRunningTime="2026-10-30 19:21:14.852390979 +0000 UTC m=+856.849306362"
Oct 30 19:21:15.254196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:15.254125    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:21:15.844193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:15.844152    6777 generic.go:356] "Generic (PLEG): container finished" podID="c9fd068c-11fc-4e31-b07b-c7653cbeb71c" containerID="8aa953bfe0f9c425166ec64e0a2aa0316b038f77f9b0bc06a48b4a5a05446bd8" exitCode=0
Oct 30 19:21:15.844418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:15.844237    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj" event={"ID":"c9fd068c-11fc-4e31-b07b-c7653cbeb71c","Type":"ContainerDied","Data":"8aa953bfe0f9c425166ec64e0a2aa0316b038f77f9b0bc06a48b4a5a05446bd8"}
Oct 30 19:21:17.518099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.490841    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:17.702132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.701770    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-kube-api-access-gjrgk\" (UniqueName: \"kubernetes.io/projected/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-kube-api-access-gjrgk\") pod \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") "
Oct 30 19:21:17.702132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.701925    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-openshift-ca\") pod \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") "
Oct 30 19:21:17.703864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.702601    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-openshift-ca" pod "c9fd068c-11fc-4e31-b07b-c7653cbeb71c" (UID: "c9fd068c-11fc-4e31-b07b-c7653cbeb71c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:17.703864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.702211    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-xks-ca\") pod \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\" (UID: \"c9fd068c-11fc-4e31-b07b-c7653cbeb71c\") "
Oct 30 19:21:17.703864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.703808    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:17.705247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.705219    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-kube-api-access-gjrgk" pod "c9fd068c-11fc-4e31-b07b-c7653cbeb71c" (UID: "c9fd068c-11fc-4e31-b07b-c7653cbeb71c"). InnerVolumeSpecName "kube-api-access-gjrgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:17.705509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.705472    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-xks-ca" pod "c9fd068c-11fc-4e31-b07b-c7653cbeb71c" (UID: "c9fd068c-11fc-4e31-b07b-c7653cbeb71c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:17.805083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.804992    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:17.805083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.805043    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrgk\" (UniqueName: \"kubernetes.io/projected/c9fd068c-11fc-4e31-b07b-c7653cbeb71c-kube-api-access-gjrgk\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:17.926081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.925170    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"
Oct 30 19:21:17.926081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.925171    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj" event={"ID":"c9fd068c-11fc-4e31-b07b-c7653cbeb71c","Type":"ContainerDied","Data":"7beeeb00cc24e0a46ffbf13516e5ce380f868acf0be9be09bdd083be588a5aa3"}
Oct 30 19:21:17.926081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:17.925878    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7beeeb00cc24e0a46ffbf13516e5ce380f868acf0be9be09bdd083be588a5aa3"
Oct 30 19:21:19.586354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.586322    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"]
Oct 30 19:21:19.588044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.588016    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c9fd068c-11fc-4e31-b07b-c7653cbeb71c" containerName="revoke-keys"
Oct 30 19:21:19.588332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.588313    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fd068c-11fc-4e31-b07b-c7653cbeb71c" containerName="revoke-keys"
Oct 30 19:21:19.588452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.588442    6777 cpu_manager.go:414] "containerMap: removing container" podUID="34252ef3-3dd7-48c4-9bde-af9f27ffc3f7" containerName="revoke-keys"
Oct 30 19:21:19.588526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.588516    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34252ef3-3dd7-48c4-9bde-af9f27ffc3f7" containerName="revoke-keys"
Oct 30 19:21:19.588866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.588802    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fd068c-11fc-4e31-b07b-c7653cbeb71c" containerName="revoke-keys"
Oct 30 19:21:19.588982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.588970    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="34252ef3-3dd7-48c4-9bde-af9f27ffc3f7" containerName="revoke-keys"
Oct 30 19:21:19.594284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.594266    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.607323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.607290    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"]
Oct 30 19:21:19.648270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.648222    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cv6v\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-kube-api-access-6cv6v\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.648485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.648355    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.648485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.648385    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.648485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.648416    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.648667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.648532    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.648667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.648638    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.648776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.648670    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe14650-0968-4dec-9701-ab16d1a8b347-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.648776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.648705    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.648894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.648826    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.749970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.749934    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cv6v\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-kube-api-access-6cv6v\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.749987    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750014    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750189    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750284    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750397    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750407    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750426    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe14650-0968-4dec-9701-ab16d1a8b347-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750437    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750453    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750630    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750647    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.750812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.750789    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.751023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.751005    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe14650-0968-4dec-9701-ab16d1a8b347-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.752489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.752461    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.752912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.752889    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.762566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.762542    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.763022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.763001    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cv6v\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-kube-api-access-6cv6v\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:19.919293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:19.919208    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:20.208962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:20.208932    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"]
Oct 30 19:21:20.210445 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:20.210419    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe14650_0968_4dec_9701_ab16d1a8b347.slice/crio-fecf557bac2036d197a86f13f31a2b63ff8ae6f62ee52bae59ac19f1519b84cf WatchSource:0}: Error finding container fecf557bac2036d197a86f13f31a2b63ff8ae6f62ee52bae59ac19f1519b84cf: Status 404 returned error can't find the container with id fecf557bac2036d197a86f13f31a2b63ff8ae6f62ee52bae59ac19f1519b84cf
Oct 30 19:21:20.212882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:20.212852    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:20.212954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:20.212913    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:20.212954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:20.212941    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:20.946421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:20.946385    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" event={"ID":"8fe14650-0968-4dec-9701-ab16d1a8b347","Type":"ContainerStarted","Data":"00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f"}
Oct 30 19:21:20.946825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:20.946440    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" event={"ID":"8fe14650-0968-4dec-9701-ab16d1a8b347","Type":"ContainerStarted","Data":"fecf557bac2036d197a86f13f31a2b63ff8ae6f62ee52bae59ac19f1519b84cf"}
Oct 30 19:21:20.997621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:20.997564    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podStartSLOduration=1.997548186 podStartE2EDuration="1.997548186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:20.996923603 +0000 UTC m=+862.993838988" watchObservedRunningTime="2026-10-30 19:21:20.997548186 +0000 UTC m=+862.994463570"
Oct 30 19:21:21.679865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.679826    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"]
Oct 30 19:21:21.683992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.683970    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.694425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.694402    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"]
Oct 30 19:21:21.760390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.760354    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:21.760832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.760770    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:21:21.785147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.785106    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d755ed8-69jbt\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.785299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.785273    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8w7g\" (UniqueName: \"kubernetes.io/projected/8d646b6e-7ffa-4594-8638-365abfc3ce2d-kube-api-access-z8w7g\") pod \"maas-api-revoke-sub-2d755ed8-69jbt\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.785380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.785361    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d755ed8-69jbt\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.885968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.885936    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d755ed8-69jbt\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.886148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.886017    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8w7g\" (UniqueName: \"kubernetes.io/projected/8d646b6e-7ffa-4594-8638-365abfc3ce2d-kube-api-access-z8w7g\") pod \"maas-api-revoke-sub-2d755ed8-69jbt\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.886148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.886082    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d755ed8-69jbt\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.886769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.886748    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d755ed8-69jbt\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.888375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.888355    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d755ed8-69jbt\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.899564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.899537    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8w7g\" (UniqueName: \"kubernetes.io/projected/8d646b6e-7ffa-4594-8638-365abfc3ce2d-kube-api-access-z8w7g\") pod \"maas-api-revoke-sub-2d755ed8-69jbt\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:21.920334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.920313    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:21.921629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.921609    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:21:21.921694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.921656    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:21:21.999690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:21.999612    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:22.341360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:22.341331    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"]
Oct 30 19:21:22.343929 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:22.343891    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d646b6e_7ffa_4594_8638_365abfc3ce2d.slice/crio-af67a8478f3dd1a10c307be6401e993252101bd321c7079e96aff079cf1c4060 WatchSource:0}: Error finding container af67a8478f3dd1a10c307be6401e993252101bd321c7079e96aff079cf1c4060: Status 404 returned error can't find the container with id af67a8478f3dd1a10c307be6401e993252101bd321c7079e96aff079cf1c4060
Oct 30 19:21:22.920808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:22.920759    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:21:22.920961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:22.920824    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:21:22.993086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:22.993036    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt" event={"ID":"8d646b6e-7ffa-4594-8638-365abfc3ce2d","Type":"ContainerStarted","Data":"21fe38cf5d4dfe5836d8dd6db8f0c184bad78e311f78ee0c9b5ebc1afb116733"}
Oct 30 19:21:22.993278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:22.993137    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt" event={"ID":"8d646b6e-7ffa-4594-8638-365abfc3ce2d","Type":"ContainerStarted","Data":"af67a8478f3dd1a10c307be6401e993252101bd321c7079e96aff079cf1c4060"}
Oct 30 19:21:23.014296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:23.014250    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt" podStartSLOduration=2.014238245 podStartE2EDuration="2.014238245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:23.013020312 +0000 UTC m=+865.009935698" watchObservedRunningTime="2026-10-30 19:21:23.014238245 +0000 UTC m=+865.011153673"
Oct 30 19:21:23.919904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:23.919871    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:21:23.920115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:23.919925    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:21:24.001010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:24.000976    6777 generic.go:356] "Generic (PLEG): container finished" podID="8d646b6e-7ffa-4594-8638-365abfc3ce2d" containerID="21fe38cf5d4dfe5836d8dd6db8f0c184bad78e311f78ee0c9b5ebc1afb116733" exitCode=0
Oct 30 19:21:24.001224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:24.001049    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt" event={"ID":"8d646b6e-7ffa-4594-8638-365abfc3ce2d","Type":"ContainerDied","Data":"21fe38cf5d4dfe5836d8dd6db8f0c184bad78e311f78ee0c9b5ebc1afb116733"}
Oct 30 19:21:24.920408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:24.920376    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:21:24.920876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:24.920849    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:21:25.202660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.202635    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:25.273587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.273476    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-openshift-ca\") pod \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") "
Oct 30 19:21:25.273587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.273579    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-xks-ca\") pod \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") "
Oct 30 19:21:25.273848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.273766    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d646b6e-7ffa-4594-8638-365abfc3ce2d-kube-api-access-z8w7g\" (UniqueName: \"kubernetes.io/projected/8d646b6e-7ffa-4594-8638-365abfc3ce2d-kube-api-access-z8w7g\") pod \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\" (UID: \"8d646b6e-7ffa-4594-8638-365abfc3ce2d\") "
Oct 30 19:21:25.274529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.274482    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-openshift-ca" pod "8d646b6e-7ffa-4594-8638-365abfc3ce2d" (UID: "8d646b6e-7ffa-4594-8638-365abfc3ce2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:25.277010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.276974    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d646b6e-7ffa-4594-8638-365abfc3ce2d-kube-api-access-z8w7g" pod "8d646b6e-7ffa-4594-8638-365abfc3ce2d" (UID: "8d646b6e-7ffa-4594-8638-365abfc3ce2d"). InnerVolumeSpecName "kube-api-access-z8w7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:25.277244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.277104    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-xks-ca" pod "8d646b6e-7ffa-4594-8638-365abfc3ce2d" (UID: "8d646b6e-7ffa-4594-8638-365abfc3ce2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:25.376104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.376035    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:25.376104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.376089    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d646b6e-7ffa-4594-8638-365abfc3ce2d-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:25.376104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.376104    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8w7g\" (UniqueName: \"kubernetes.io/projected/8d646b6e-7ffa-4594-8638-365abfc3ce2d-kube-api-access-z8w7g\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:25.919762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.919728    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:21:25.919955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:25.919781    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:21:26.015810 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:26.015774    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"
Oct 30 19:21:26.015810 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:26.015784    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt" event={"ID":"8d646b6e-7ffa-4594-8638-365abfc3ce2d","Type":"ContainerDied","Data":"af67a8478f3dd1a10c307be6401e993252101bd321c7079e96aff079cf1c4060"}
Oct 30 19:21:26.016363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:26.015826    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af67a8478f3dd1a10c307be6401e993252101bd321c7079e96aff079cf1c4060"
Oct 30 19:21:26.924405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:26.924375    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:26.924683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:26.924661    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:26.925277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:26.925254    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:21:32.382272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:32.382228    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-chjvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:21:32.382874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:32.382837    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:21:32.424204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:32.424167    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:21:32.424442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:32.424421    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:21:34.566338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.566299    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"]
Oct 30 19:21:34.567492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.567458    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8d646b6e-7ffa-4594-8638-365abfc3ce2d" containerName="revoke-keys"
Oct 30 19:21:34.567492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.567482    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d646b6e-7ffa-4594-8638-365abfc3ce2d" containerName="revoke-keys"
Oct 30 19:21:34.567694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.567605    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d646b6e-7ffa-4594-8638-365abfc3ce2d" containerName="revoke-keys"
Oct 30 19:21:34.572315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.572282    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.578179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.578152    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:21:34.578718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.578698    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:34.587926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.587894    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"]
Oct 30 19:21:34.668278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.668236    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-636e43b6-wctbg\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.668518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.668444    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7r4\" (UniqueName: \"kubernetes.io/projected/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-kube-api-access-4x7r4\") pod \"maas-api-revoke-sub-636e43b6-wctbg\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.668923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.668901    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-636e43b6-wctbg\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.682622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.682594    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"]
Oct 30 19:21:34.686784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.686762    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.702478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.702449    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"]
Oct 30 19:21:34.770012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.769972    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7r4\" (UniqueName: \"kubernetes.io/projected/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-kube-api-access-4x7r4\") pod \"maas-api-revoke-sub-636e43b6-wctbg\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.770220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770024    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.770220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770047    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-podinfo\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.770337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770221    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-certs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.770394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770336    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-636e43b6-wctbg\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.770394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770380    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-credential-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.770490 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770445    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-token\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.770528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770490    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-data\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.770620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770601    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ded767-98c6-459e-9209-3c58dc4751c8-istiod-ca-cert\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.770666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770652    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-636e43b6-wctbg\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.770713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770704    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-envoy\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.770762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.770725    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqhm8\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-kube-api-access-hqhm8\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.771318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.771295    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-636e43b6-wctbg\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.772716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.772696    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-636e43b6-wctbg\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.782918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.782897    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7r4\" (UniqueName: \"kubernetes.io/projected/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-kube-api-access-4x7r4\") pod \"maas-api-revoke-sub-636e43b6-wctbg\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:34.872075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.871981    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872019    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-podinfo\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872069    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-certs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872107    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-credential-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872249    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-token\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872321    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-data\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872382    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ded767-98c6-459e-9209-3c58dc4751c8-istiod-ca-cert\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872438    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872456    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-envoy\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872485    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-certs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872502    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqhm8\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-kube-api-access-hqhm8\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872510    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-credential-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.872767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.872677    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-data\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.873027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.873007    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ded767-98c6-459e-9209-3c58dc4751c8-istiod-ca-cert\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.874491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.874464    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-envoy\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.874601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.874551    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-podinfo\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.887973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.887940    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqhm8\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-kube-api-access-hqhm8\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.888584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.888566    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-token\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:34.896681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:34.896659    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:35.004632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:35.004603    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:35.175376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:35.175347    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"]
Oct 30 19:21:35.176014 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:35.175990    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cadbd8_4940_4aa9_8eb7_52b845ff4aeb.slice/crio-b545b7e027008a6ccb0e1dfd207f8d2837135fd43fda799b507b9e9bff76430a WatchSource:0}: Error finding container b545b7e027008a6ccb0e1dfd207f8d2837135fd43fda799b507b9e9bff76430a: Status 404 returned error can't find the container with id b545b7e027008a6ccb0e1dfd207f8d2837135fd43fda799b507b9e9bff76430a
Oct 30 19:21:35.299010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:35.298981    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"]
Oct 30 19:21:35.299536 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:35.299507    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ded767_98c6_459e_9209_3c58dc4751c8.slice/crio-2eec1a5d2b562f02ce02b78e113a617b341d2536dfbbf9f9d3cd2ec431912b88 WatchSource:0}: Error finding container 2eec1a5d2b562f02ce02b78e113a617b341d2536dfbbf9f9d3cd2ec431912b88: Status 404 returned error can't find the container with id 2eec1a5d2b562f02ce02b78e113a617b341d2536dfbbf9f9d3cd2ec431912b88
Oct 30 19:21:35.302010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:35.301980    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:35.302138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:35.302044    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:35.302138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:35.302106    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:36.199670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:36.199635    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" event={"ID":"b7ded767-98c6-459e-9209-3c58dc4751c8","Type":"ContainerStarted","Data":"0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c"}
Oct 30 19:21:36.200363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:36.199695    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" event={"ID":"b7ded767-98c6-459e-9209-3c58dc4751c8","Type":"ContainerStarted","Data":"2eec1a5d2b562f02ce02b78e113a617b341d2536dfbbf9f9d3cd2ec431912b88"}
Oct 30 19:21:36.202517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:36.202486    6777 generic.go:356] "Generic (PLEG): container finished" podID="c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb" containerID="b57cc5b6d317367006f208849397dc341edcced02f947ae7a4b7d2a08fc1d2c2" exitCode=0
Oct 30 19:21:36.202688 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:36.202564    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg" event={"ID":"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb","Type":"ContainerDied","Data":"b57cc5b6d317367006f208849397dc341edcced02f947ae7a4b7d2a08fc1d2c2"}
Oct 30 19:21:36.202688 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:36.202611    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg" event={"ID":"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb","Type":"ContainerStarted","Data":"b545b7e027008a6ccb0e1dfd207f8d2837135fd43fda799b507b9e9bff76430a"}
Oct 30 19:21:36.226965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:36.226861    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" podStartSLOduration=2.22684771 podStartE2EDuration="2.22684771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:36.226496366 +0000 UTC m=+878.223411748" watchObservedRunningTime="2026-10-30 19:21:36.22684771 +0000 UTC m=+878.223763093"
Oct 30 19:21:37.005013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.004979    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:37.006358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.006330    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:37.006509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.006381    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:21:37.422653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.422623    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:37.510529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.510490    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-xks-ca\") pod \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") "
Oct 30 19:21:37.510696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.510596    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-kube-api-access-4x7r4\" (UniqueName: \"kubernetes.io/projected/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-kube-api-access-4x7r4\") pod \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") "
Oct 30 19:21:37.510737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.510698    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-openshift-ca\") pod \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\" (UID: \"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb\") "
Oct 30 19:21:37.511129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.511096    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-openshift-ca" pod "c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb" (UID: "c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:37.512562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.512539    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-xks-ca" pod "c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb" (UID: "c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:37.512686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.512667    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-kube-api-access-4x7r4" pod "c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb" (UID: "c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb"). InnerVolumeSpecName "kube-api-access-4x7r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:37.612116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.612000    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7r4\" (UniqueName: \"kubernetes.io/projected/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-kube-api-access-4x7r4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:37.612116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.612033    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:37.612116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.612044    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:37.852170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.848016    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"]
Oct 30 19:21:37.852170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.848878    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb" containerName="revoke-keys"
Oct 30 19:21:37.852170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.848897    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb" containerName="revoke-keys"
Oct 30 19:21:37.856090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.849052    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb" containerName="revoke-keys"
Oct 30 19:21:37.873357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.868555    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:37.913908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.913742    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"]
Oct 30 19:21:37.915128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.915101    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:37.915255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.915138    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:37.915255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.915158    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzg79\" (UniqueName: \"kubernetes.io/projected/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-kube-api-access-wzg79\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:37.915255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.915176    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:37.915367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.915346    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:37.915423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.915409    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:37.915511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.915444    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:37.915664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.915537    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:37.915664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:37.915633    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.005145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.005111    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:38.005319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.005174    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:21:38.016927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.016898    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.016927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.016930    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.016978    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017070    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017119    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017146    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017235    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzg79\" (UniqueName: \"kubernetes.io/projected/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-kube-api-access-wzg79\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017292    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017323    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017397    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017402    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017505    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017619    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.017791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.017771    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.019565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.019536    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.019678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.019639    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.033021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.033002    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.033188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.033171    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzg79\" (UniqueName: \"kubernetes.io/projected/c5a9e30d-41c9-45a4-a6ce-396e5d3f8952-kube-api-access-wzg79\") pod \"maas-default-gateway-openshift-default-855f9574cf-wz7wl\" (UID: \"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.217456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.217374    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"
Oct 30 19:21:38.217456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.217405    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg" event={"ID":"c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb","Type":"ContainerDied","Data":"b545b7e027008a6ccb0e1dfd207f8d2837135fd43fda799b507b9e9bff76430a"}
Oct 30 19:21:38.217671 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.217456    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b545b7e027008a6ccb0e1dfd207f8d2837135fd43fda799b507b9e9bff76430a"
Oct 30 19:21:38.225925 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.225900    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:38.506252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.506175    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:21:38.579511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.579282    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"]
Oct 30 19:21:38.590375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.587577    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:38.590375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.587656    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:38.590375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:38.587701    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:39.005631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:39.005596    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:39.005791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:39.005659    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:21:39.226978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:39.226944    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" event={"ID":"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952","Type":"ContainerStarted","Data":"ed0d4532cbff5eceb2a090f8f2e23392265476775dea2a93b6eb2188a7771154"}
Oct 30 19:21:39.227216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:39.227004    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" event={"ID":"c5a9e30d-41c9-45a4-a6ce-396e5d3f8952","Type":"ContainerStarted","Data":"719084b3f4f1e7d380835f87355923e2ea7f8700f174362b75f7aedfb896154d"}
Oct 30 19:21:39.256144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:39.253618    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" podStartSLOduration=2.253597102 podStartE2EDuration="2.253597102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:39.252379785 +0000 UTC m=+881.249295187" watchObservedRunningTime="2026-10-30 19:21:39.253597102 +0000 UTC m=+881.250512487"
Oct 30 19:21:40.020082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:40.013171    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:40.020082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:40.013245    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:21:40.232085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:40.231370    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:40.236082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:40.233282    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wz7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:40.236082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:40.233371    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" podUID="c5a9e30d-41c9-45a4-a6ce-396e5d3f8952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:21:41.014083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:41.010449    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:41.014083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:41.010507    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:21:41.022085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:41.020749    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:21:41.054083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:41.053158    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:21:41.094942 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:41.094264    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:21:41.222082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:41.221252    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:21:41.229746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:41.229166    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wz7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:41.229746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:41.229227    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" podUID="c5a9e30d-41c9-45a4-a6ce-396e5d3f8952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:21:42.005822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:42.005787    6777 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:42.006016 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:42.005843    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:21:42.227291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:42.227256    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wz7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:42.227775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:42.227318    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" podUID="c5a9e30d-41c9-45a4-a6ce-396e5d3f8952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:21:43.009805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:43.009771    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:43.010094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:43.010043    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:43.010860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:43.010844    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:21:43.227168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:43.227051    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wz7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:43.227168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:43.227138    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" podUID="c5a9e30d-41c9-45a4-a6ce-396e5d3f8952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:21:43.640358 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:21:43.640303    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:21:44.227337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:44.227303    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wz7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:44.227832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:44.227361    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" podUID="c5a9e30d-41c9-45a4-a6ce-396e5d3f8952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:21:45.226327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:45.226296    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wz7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:45.226503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:45.226349    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" podUID="c5a9e30d-41c9-45a4-a6ce-396e5d3f8952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:21:46.226911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.226870    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wz7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:46.227580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.227532    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl" podUID="c5a9e30d-41c9-45a4-a6ce-396e5d3f8952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:21:46.594210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.594173    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"]
Oct 30 19:21:46.598276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.598255    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.611243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.611213    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"]
Oct 30 19:21:46.612084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.612039    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"distinct-2-w0-ad23c7-kserve-self-signed-certs\""
Oct 30 19:21:46.768041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.765533    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-dshm\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.768041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.765723    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-model-cache\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.768041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.765810    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b71be04-6fbe-4637-a36b-ff16915906f2-tls-certs\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.768041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.765935    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5z5g\" (UniqueName: \"kubernetes.io/projected/3b71be04-6fbe-4637-a36b-ff16915906f2-kube-api-access-z5z5g\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.768041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.766001    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-tmp-dir\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.768041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.766027    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-home\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.867419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.867329    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-model-cache\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.867419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.867396    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b71be04-6fbe-4637-a36b-ff16915906f2-tls-certs\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.867656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.867447    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5z5g\" (UniqueName: \"kubernetes.io/projected/3b71be04-6fbe-4637-a36b-ff16915906f2-kube-api-access-z5z5g\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.867656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.867494    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-tmp-dir\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.867656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.867511    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-home\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.867656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.867550    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-dshm\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.867888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.867839    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-model-cache\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.867995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.867960    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-tmp-dir\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.868336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.868311    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-home\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.870208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.870187    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-dshm\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.870308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.870186    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b71be04-6fbe-4637-a36b-ff16915906f2-tls-certs\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.880427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.880399    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5z5g\" (UniqueName: \"kubernetes.io/projected/3b71be04-6fbe-4637-a36b-ff16915906f2-kube-api-access-z5z5g\") pod \"distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:46.917317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:46.917280    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:47.214825 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:47.214789    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b71be04_6fbe_4637_a36b_ff16915906f2.slice/crio-5a88dc6d16228775c94235a855a8a566f51ce56e25fc1e0f9573a48210facf07 WatchSource:0}: Error finding container 5a88dc6d16228775c94235a855a8a566f51ce56e25fc1e0f9573a48210facf07: Status 404 returned error can't find the container with id 5a88dc6d16228775c94235a855a8a566f51ce56e25fc1e0f9573a48210facf07
Oct 30 19:21:47.215495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:47.215467    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"]
Oct 30 19:21:47.358546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:47.358492    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" event={"ID":"3b71be04-6fbe-4637-a36b-ff16915906f2","Type":"ContainerStarted","Data":"dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651"}
Oct 30 19:21:47.359080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:47.358554    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" event={"ID":"3b71be04-6fbe-4637-a36b-ff16915906f2","Type":"ContainerStarted","Data":"5a88dc6d16228775c94235a855a8a566f51ce56e25fc1e0f9573a48210facf07"}
Oct 30 19:21:47.388543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:47.388478    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podStartSLOduration=1.388457633 podStartE2EDuration="1.388457633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:47.382388614 +0000 UTC m=+889.379304004" watchObservedRunningTime="2026-10-30 19:21:47.388457633 +0000 UTC m=+889.385373017"
Oct 30 19:21:47.581532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:47.581496    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:47.582088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:47.582047    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:47.582958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:47.582932    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wz7wl"
Oct 30 19:21:47.917955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:47.917877    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:49.850525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.850489    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"]
Oct 30 19:21:49.854905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.854881    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:49.874096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.874040    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"]
Oct 30 19:21:49.914910 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.914878    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:49.915237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.914956    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:49.915237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.915178    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:49.915237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.915231    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a618ff-60dc-4c04-aaca-1f40b343a57e-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:49.915462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.915288    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkvp5\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-kube-api-access-mkvp5\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:49.915462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.915337    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:49.915462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.915383    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:49.915640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.915520    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:49.915640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:49.915566    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016456    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016545    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016589    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016612    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a618ff-60dc-4c04-aaca-1f40b343a57e-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016638    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkvp5\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-kube-api-access-mkvp5\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016668    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016707    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016835    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016898    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.016960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.016930    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.017260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.017229    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.017383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.017258    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a618ff-60dc-4c04-aaca-1f40b343a57e-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.017383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.017325    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.017383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.017357    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.019352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.019329    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.019546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.019527    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.029114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.029093    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.029245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.029227    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkvp5\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-kube-api-access-mkvp5\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.176446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.176362    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:50.178341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.178315    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"]
Oct 30 19:21:50.182449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.182420    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.201335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.201305    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"]
Oct 30 19:21:50.219622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.219573    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.219802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.219630    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6449741c-acf1-41b6-80dd-e3416a04a548-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.219802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.219703    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.219949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.219814    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.219949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.219838    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88p9\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-kube-api-access-d88p9\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.219949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.219889    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.219949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.219920    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.220177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.219956    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.220538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.220487    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6449741c-acf1-41b6-80dd-e3416a04a548-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.321608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.321565    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6449741c-acf1-41b6-80dd-e3416a04a548-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.321811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.321657    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.321811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.321687    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6449741c-acf1-41b6-80dd-e3416a04a548-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.321811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.321737    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.321985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.321843    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.321985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.321871    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88p9\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-kube-api-access-d88p9\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.321985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.321923    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.321985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.321951    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.322236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.321987    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.322366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.322340    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.322441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.322404    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.322441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.322423    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6449741c-acf1-41b6-80dd-e3416a04a548-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.322665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.322638    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.322830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.322809    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.324170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.324150    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.324818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.324798    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6449741c-acf1-41b6-80dd-e3416a04a548-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.334102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.334073    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.334241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.334224    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88p9\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-kube-api-access-d88p9\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.485271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.485238    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"]
Oct 30 19:21:50.485971 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:50.485944    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a618ff_60dc_4c04_aaca_1f40b343a57e.slice/crio-3e43ac46ac2a10d2a3b58f504b39a209d056084d9f9e6da2ffd4ec5e245ac4d7 WatchSource:0}: Error finding container 3e43ac46ac2a10d2a3b58f504b39a209d056084d9f9e6da2ffd4ec5e245ac4d7: Status 404 returned error can't find the container with id 3e43ac46ac2a10d2a3b58f504b39a209d056084d9f9e6da2ffd4ec5e245ac4d7
Oct 30 19:21:50.488957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.488922    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:50.489040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.488990    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:50.489040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.489021    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:50.528965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.528942    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:50.817651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.817621    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"]
Oct 30 19:21:50.818870 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:21:50.818845    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6449741c_acf1_41b6_80dd_e3416a04a548.slice/crio-4e824bd9650dda3029b7b34dc2ee74c268c4f7b0b04aaf74fdeb32789b2427ca WatchSource:0}: Error finding container 4e824bd9650dda3029b7b34dc2ee74c268c4f7b0b04aaf74fdeb32789b2427ca: Status 404 returned error can't find the container with id 4e824bd9650dda3029b7b34dc2ee74c268c4f7b0b04aaf74fdeb32789b2427ca
Oct 30 19:21:50.822023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.821992    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:50.822208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.822186    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:50.822282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:50.822250    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:21:51.386048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:51.386004    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" event={"ID":"89a618ff-60dc-4c04-aaca-1f40b343a57e","Type":"ContainerStarted","Data":"8f09d5ca7a902aedc7a8a9728a33624d0cf698b94b2cc39db08e7813a4cde566"}
Oct 30 19:21:51.386648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:51.386126    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" event={"ID":"89a618ff-60dc-4c04-aaca-1f40b343a57e","Type":"ContainerStarted","Data":"3e43ac46ac2a10d2a3b58f504b39a209d056084d9f9e6da2ffd4ec5e245ac4d7"}
Oct 30 19:21:51.387867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:51.387843    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" event={"ID":"6449741c-acf1-41b6-80dd-e3416a04a548","Type":"ContainerStarted","Data":"5bce58a4a5b13594139691ffaaa21d8d081a6aad30f422bae98ae9fb23265142"}
Oct 30 19:21:51.388020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:51.387892    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" event={"ID":"6449741c-acf1-41b6-80dd-e3416a04a548","Type":"ContainerStarted","Data":"4e824bd9650dda3029b7b34dc2ee74c268c4f7b0b04aaf74fdeb32789b2427ca"}
Oct 30 19:21:51.413592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:51.413547    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" podStartSLOduration=2.413531455 podStartE2EDuration="2.413531455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:51.412431445 +0000 UTC m=+893.409346837" watchObservedRunningTime="2026-10-30 19:21:51.413531455 +0000 UTC m=+893.410446839"
Oct 30 19:21:51.442078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:51.442018    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podStartSLOduration=1.442004017 podStartE2EDuration="1.442004017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:51.441158187 +0000 UTC m=+893.438073571" watchObservedRunningTime="2026-10-30 19:21:51.442004017 +0000 UTC m=+893.438919400"
Oct 30 19:21:51.529844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:51.529804    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:21:51.531351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:51.531322    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:21:51.531476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:51.531371    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 19:21:52.189084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:52.179431    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:52.211088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:52.193200    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:21:52.211088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:52.193267    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:21:52.529800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:52.529769    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:21:52.530235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:52.529836    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 19:21:52.980105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:52.979994    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"]
Oct 30 19:21:53.018325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.017137    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.023100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.023006    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"]
Oct 30 19:21:53.024761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.024742    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2d5d53\"/\"openshift-service-ca.crt\""
Oct 30 19:21:53.025095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.025053    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2d5d53\"/\"kube-root-ca.crt\""
Oct 30 19:21:53.025197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.025183    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2d5d53\"/\"default-dockercfg-8rj6n\""
Oct 30 19:21:53.025281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.025266    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2d5d53\"/\"distinct-2-w5-2d5d53-kserve-self-signed-certs\""
Oct 30 19:21:53.108613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.108565    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-tmp-dir\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.108808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.108746    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-dshm\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.108958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.108930    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64afb4d-4a06-410b-b249-d7fdeccb0226-tls-certs\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.109044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.108971    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-home\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.109044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.109006    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-model-cache\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.109192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.109081    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6tw\" (UniqueName: \"kubernetes.io/projected/f64afb4d-4a06-410b-b249-d7fdeccb0226-kube-api-access-8t6tw\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.177459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.177420    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:21:53.177794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.177755    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:21:53.211929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.211880    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-home\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.212205 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.212179    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-model-cache\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.212372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.212354    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6tw\" (UniqueName: \"kubernetes.io/projected/f64afb4d-4a06-410b-b249-d7fdeccb0226-kube-api-access-8t6tw\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.212581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.212560    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-tmp-dir\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.212778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.212752    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-dshm\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.219084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.213276    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64afb4d-4a06-410b-b249-d7fdeccb0226-tls-certs\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.224287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.224256    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-home\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.224772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.224748    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-model-cache\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.245734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.245654    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-tmp-dir\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.258199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.256871    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64afb4d-4a06-410b-b249-d7fdeccb0226-tls-certs\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.275909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.275877    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-dshm\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.303554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.303484    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6tw\" (UniqueName: \"kubernetes.io/projected/f64afb4d-4a06-410b-b249-d7fdeccb0226-kube-api-access-8t6tw\") pod \"distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") " pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.336189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.336159    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:53.529763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.529739    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:21:53.529908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.529789    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 19:21:53.704727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:53.704559    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"]
Oct 30 19:21:54.180580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:54.180481    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:21:54.180852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:54.180818    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:21:54.416415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:54.416377    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" event={"ID":"f64afb4d-4a06-410b-b249-d7fdeccb0226","Type":"ContainerStarted","Data":"7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0"}
Oct 30 19:21:54.416608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:54.416426    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" event={"ID":"f64afb4d-4a06-410b-b249-d7fdeccb0226","Type":"ContainerStarted","Data":"4fc450843f2cd643595b96ba6fc07e9c8d7fe2f5a01a066d98ee5b36f034d51c"}
Oct 30 19:21:54.436968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:54.436865    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podStartSLOduration=2.43684925 podStartE2EDuration="2.43684925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:54.436540007 +0000 UTC m=+896.433455388" watchObservedRunningTime="2026-10-30 19:21:54.43684925 +0000 UTC m=+896.433764635"
Oct 30 19:21:54.529863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:54.529827    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:21:54.530349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:54.530047    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 19:21:55.187085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:55.185479    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:21:55.187085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:55.185544    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:21:55.336709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:55.336673    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:21:55.530402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:55.530372    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:21:55.530827 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:55.530801    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 19:21:56.185082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:56.184523    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:21:56.185082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:56.184609    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:21:56.529584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:56.529549    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:21:56.529854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:56.529819    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 19:21:56.923141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:56.921749    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:57.162080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:57.161683    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:57.185136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:57.179438    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:21:57.185136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:57.179498    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:21:57.492149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:57.483996    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:21:57.537084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:57.536482    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:21:57.537084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:57.536564    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 19:21:58.178082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:58.177119    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:21:58.178082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:58.177189    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:21:58.534080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:58.532267    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:21:58.534080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:58.532330    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 19:21:59.197083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:59.196400    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:59.201081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:59.197939    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:21:59.210088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:21:59.207512    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:22:00.526695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.526656    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"]
Oct 30 19:22:00.532414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.532390    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.539627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.539601    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:22:00.540331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.540142    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:00.545568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.545541    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"]
Oct 30 19:22:00.582559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.582511    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:22:00.582757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.582590    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:00.620486 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.620448    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c0b735e-2xsg5\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.620486 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.620489    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6nk\" (UniqueName: \"kubernetes.io/projected/beae642a-7921-4f8d-a903-7336dba1dc2a-kube-api-access-9d6nk\") pod \"maas-api-revoke-sub-4c0b735e-2xsg5\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.620699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.620685    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c0b735e-2xsg5\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.722564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.722526    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c0b735e-2xsg5\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.722778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.722733    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c0b735e-2xsg5\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.722778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.722755    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6nk\" (UniqueName: \"kubernetes.io/projected/beae642a-7921-4f8d-a903-7336dba1dc2a-kube-api-access-9d6nk\") pod \"maas-api-revoke-sub-4c0b735e-2xsg5\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.723281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.723257    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c0b735e-2xsg5\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.725053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.725030    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c0b735e-2xsg5\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.735760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.735734    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6nk\" (UniqueName: \"kubernetes.io/projected/beae642a-7921-4f8d-a903-7336dba1dc2a-kube-api-access-9d6nk\") pod \"maas-api-revoke-sub-4c0b735e-2xsg5\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.848439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.848398    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:00.867536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.867509    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:22:00.869010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.868985    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:22:00.873894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:00.873866    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:22:01.147310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:01.147280    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"]
Oct 30 19:22:01.147792 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:22:01.147767    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeae642a_7921_4f8d_a903_7336dba1dc2a.slice/crio-af4202489ed1b0ff81ec30236f08598cda9805995c23dfcc17911a4fe4ad402b WatchSource:0}: Error finding container af4202489ed1b0ff81ec30236f08598cda9805995c23dfcc17911a4fe4ad402b: Status 404 returned error can't find the container with id af4202489ed1b0ff81ec30236f08598cda9805995c23dfcc17911a4fe4ad402b
Oct 30 19:22:01.578314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:01.578273    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5" event={"ID":"beae642a-7921-4f8d-a903-7336dba1dc2a","Type":"ContainerStarted","Data":"75e1418a9369c04aaab330785ce954e423bb01263cc0780375f44fd51efad5d9"}
Oct 30 19:22:01.578785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:01.578366    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5" event={"ID":"beae642a-7921-4f8d-a903-7336dba1dc2a","Type":"ContainerStarted","Data":"af4202489ed1b0ff81ec30236f08598cda9805995c23dfcc17911a4fe4ad402b"}
Oct 30 19:22:01.600408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:01.600346    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5" podStartSLOduration=1.600329246 podStartE2EDuration="1.600329246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:01.597732391 +0000 UTC m=+903.594647777" watchObservedRunningTime="2026-10-30 19:22:01.600329246 +0000 UTC m=+903.597244632"
Oct 30 19:22:02.587234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:02.587198    6777 generic.go:356] "Generic (PLEG): container finished" podID="beae642a-7921-4f8d-a903-7336dba1dc2a" containerID="75e1418a9369c04aaab330785ce954e423bb01263cc0780375f44fd51efad5d9" exitCode=0
Oct 30 19:22:02.587692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:02.587279    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5" event={"ID":"beae642a-7921-4f8d-a903-7336dba1dc2a","Type":"ContainerDied","Data":"75e1418a9369c04aaab330785ce954e423bb01263cc0780375f44fd51efad5d9"}
Oct 30 19:22:03.336783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.336744    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:22:03.349667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.349631    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:22:03.604885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.604859    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:22:03.772074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.772038    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:03.867966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.867878    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beae642a-7921-4f8d-a903-7336dba1dc2a-kube-api-access-9d6nk\" (UniqueName: \"kubernetes.io/projected/beae642a-7921-4f8d-a903-7336dba1dc2a-kube-api-access-9d6nk\") pod \"beae642a-7921-4f8d-a903-7336dba1dc2a\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") "
Oct 30 19:22:03.867966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.867929    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-xks-ca\") pod \"beae642a-7921-4f8d-a903-7336dba1dc2a\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") "
Oct 30 19:22:03.867966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.867966    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-openshift-ca\") pod \"beae642a-7921-4f8d-a903-7336dba1dc2a\" (UID: \"beae642a-7921-4f8d-a903-7336dba1dc2a\") "
Oct 30 19:22:03.868414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.868392    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-openshift-ca" pod "beae642a-7921-4f8d-a903-7336dba1dc2a" (UID: "beae642a-7921-4f8d-a903-7336dba1dc2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:03.868566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.868551    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:03.870123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.870040    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-xks-ca" pod "beae642a-7921-4f8d-a903-7336dba1dc2a" (UID: "beae642a-7921-4f8d-a903-7336dba1dc2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:03.870123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.870115    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beae642a-7921-4f8d-a903-7336dba1dc2a-kube-api-access-9d6nk" pod "beae642a-7921-4f8d-a903-7336dba1dc2a" (UID: "beae642a-7921-4f8d-a903-7336dba1dc2a"). InnerVolumeSpecName "kube-api-access-9d6nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:03.970141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.970112    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6nk\" (UniqueName: \"kubernetes.io/projected/beae642a-7921-4f8d-a903-7336dba1dc2a-kube-api-access-9d6nk\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:03.970141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:03.970137    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beae642a-7921-4f8d-a903-7336dba1dc2a-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:04.601582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:04.601550    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"
Oct 30 19:22:04.601747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:04.601580    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5" event={"ID":"beae642a-7921-4f8d-a903-7336dba1dc2a","Type":"ContainerDied","Data":"af4202489ed1b0ff81ec30236f08598cda9805995c23dfcc17911a4fe4ad402b"}
Oct 30 19:22:04.601747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:04.601634    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4202489ed1b0ff81ec30236f08598cda9805995c23dfcc17911a4fe4ad402b"
Oct 30 19:22:05.081603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.081573    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"]
Oct 30 19:22:05.082431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.082406    6777 cpu_manager.go:414] "containerMap: removing container" podUID="beae642a-7921-4f8d-a903-7336dba1dc2a" containerName="revoke-keys"
Oct 30 19:22:05.082431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.082430    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beae642a-7921-4f8d-a903-7336dba1dc2a" containerName="revoke-keys"
Oct 30 19:22:05.082627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.082612    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="beae642a-7921-4f8d-a903-7336dba1dc2a" containerName="revoke-keys"
Oct 30 19:22:05.086577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.086554    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.102579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.102552    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"]
Oct 30 19:22:05.185440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.185405    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.185588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.185461    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.185588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.185532    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.185761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.185739    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.185853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.185840    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwq8m\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-kube-api-access-mwq8m\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.185956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.185941    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.185992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.185969    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.186081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.186044    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.186138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.186084    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.287480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287441    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.287661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287527    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwq8m\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-kube-api-access-mwq8m\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.287661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287587    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.287661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287623    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.287820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287660    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.287820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287685    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.287820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287796    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.287978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287854    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.287978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287909    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.288124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.287898    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.288124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.288077    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.288312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.288282    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.288472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.288388    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.288472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.288446    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.290605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.290582    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.291500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.291479    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.293645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.293623    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"]
Oct 30 19:22:05.304609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.303775    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwq8m\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-kube-api-access-mwq8m\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.304609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.303881    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.307521 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.305361    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.313294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.313266    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"]
Oct 30 19:22:05.389092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.388998    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.389092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.389049    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.389289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.389093    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.389289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.389214    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.389388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.389358    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.389432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.389405    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.389478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.389449    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrm8\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-kube-api-access-8qrm8\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.389533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.389518    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ca541-7f88-46cd-aa63-beb3988e72cd-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.389583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.389574    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.403133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.403104    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.491116    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.491237    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.491297    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.491325    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.491389    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.491527    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.491567    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.491642    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrm8\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-kube-api-access-8qrm8\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.491704    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ca541-7f88-46cd-aa63-beb3988e72cd-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.492582    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ca541-7f88-46cd-aa63-beb3988e72cd-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.493683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.493654    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.494136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.494116    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.494261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.494240    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.494392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.494371    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.496131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.496106    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.496523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.496505    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.505701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.505678    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.513587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.513562    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrm8\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-kube-api-access-8qrm8\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.642822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.642741    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:05.720843 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:22:05.720798    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad79567_24bd_4ed5_9be8_59c3a09b9e6d.slice/crio-a36dfa45fb226b2d5a1d26fd4bc003c3e0d34c8309370cbf3f9ffe2d06279fe7 WatchSource:0}: Error finding container a36dfa45fb226b2d5a1d26fd4bc003c3e0d34c8309370cbf3f9ffe2d06279fe7: Status 404 returned error can't find the container with id a36dfa45fb226b2d5a1d26fd4bc003c3e0d34c8309370cbf3f9ffe2d06279fe7
Oct 30 19:22:05.722154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.722131    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"]
Oct 30 19:22:05.723533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.723495    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:05.723632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.723577    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:05.723632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.723606    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:05.926743 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:22:05.926708    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56ca541_7f88_46cd_aa63_beb3988e72cd.slice/crio-dbaaf7896543a31a6b66e47d1b1fb317abf8d7aa01867fd35b6295864ea2852e WatchSource:0}: Error finding container dbaaf7896543a31a6b66e47d1b1fb317abf8d7aa01867fd35b6295864ea2852e: Status 404 returned error can't find the container with id dbaaf7896543a31a6b66e47d1b1fb317abf8d7aa01867fd35b6295864ea2852e
Oct 30 19:22:05.927636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.927603    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"]
Oct 30 19:22:05.929560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.929529    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:05.929648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.929595    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:05.929648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:05.929621    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:06.618473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:06.618434    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" event={"ID":"bad79567-24bd-4ed5-9be8-59c3a09b9e6d","Type":"ContainerStarted","Data":"c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9"}
Oct 30 19:22:06.618963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:06.618532    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" event={"ID":"bad79567-24bd-4ed5-9be8-59c3a09b9e6d","Type":"ContainerStarted","Data":"a36dfa45fb226b2d5a1d26fd4bc003c3e0d34c8309370cbf3f9ffe2d06279fe7"}
Oct 30 19:22:06.620092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:06.620036    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" event={"ID":"b56ca541-7f88-46cd-aa63-beb3988e72cd","Type":"ContainerStarted","Data":"a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265"}
Oct 30 19:22:06.620232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:06.620101    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" event={"ID":"b56ca541-7f88-46cd-aa63-beb3988e72cd","Type":"ContainerStarted","Data":"dbaaf7896543a31a6b66e47d1b1fb317abf8d7aa01867fd35b6295864ea2852e"}
Oct 30 19:22:06.643959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:06.643935    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:06.646101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:06.646034    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podStartSLOduration=1.6460209620000001 podStartE2EDuration="1.646020962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:06.643498775 +0000 UTC m=+908.640414161" watchObservedRunningTime="2026-10-30 19:22:06.646020962 +0000 UTC m=+908.642936345"
Oct 30 19:22:06.646280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:06.646247    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:22:06.646367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:06.646322    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 19:22:06.690084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:06.684945    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" podStartSLOduration=1.684924469 podStartE2EDuration="1.684924469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:06.684363233 +0000 UTC m=+908.681278628" watchObservedRunningTime="2026-10-30 19:22:06.684924469 +0000 UTC m=+908.681839854"
Oct 30 19:22:07.403304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:07.403262    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:07.405225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:07.405195    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:07.405343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:07.405275    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 19:22:07.643620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:07.643590    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:22:07.644085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:07.644047    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 19:22:08.403678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:08.403639    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:08.403904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:08.403694    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 19:22:08.643375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:08.643341    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:22:08.643557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:08.643389    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 19:22:09.425084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:09.424836    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:09.425084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:09.424896    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 19:22:09.644579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:09.644539    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:22:09.644785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:09.644592    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 19:22:10.404132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:10.403508    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:10.404132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:10.403572    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 19:22:10.643312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:10.643276    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:22:10.643723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:10.643331    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 19:22:11.404595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:11.404561    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:11.404787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:11.404620    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 19:22:11.643592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:11.643545    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:22:11.643592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:11.643613    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 19:22:12.403565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:12.403521    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:12.403751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:12.403586    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 19:22:12.726145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:12.726046    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:12.726919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:12.726895    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:12.729734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:12.729711    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:22:13.409007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:13.408978    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:13.409316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:13.409291    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:13.410110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:13.410092    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:22:13.640108 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:22:13.640039    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:22:17.440455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:17.440413    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:22:19.726105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.726037    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"]
Oct 30 19:22:19.731255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.731227    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:19.744390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.744360    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"]
Oct 30 19:22:19.915549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.915498    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:19.915744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.915672    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:19.915896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.915880    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:19.915990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.915973    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:19.916040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.916000    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xm5\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-kube-api-access-g7xm5\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:19.916172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.916138    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:19.916326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.916188    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f1534e-4438-4789-aeee-f944b3dd15fc-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:19.916326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.916275    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:19.916441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:19.916342    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.017667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.017582    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.017667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.017663    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.017890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.017685    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xm5\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-kube-api-access-g7xm5\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.017890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.017713    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.017890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.017734    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f1534e-4438-4789-aeee-f944b3dd15fc-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.017890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.017769    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.017890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.017832    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.017890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.017884    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.018248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.017941    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.018248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.018192    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.018360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.018244    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.018594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.018568    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.018718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.018604    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.018718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.018636    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f1534e-4438-4789-aeee-f944b3dd15fc-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.020180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.020161    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.020660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.020642    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.030277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.030248    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.030356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.030297    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xm5\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-kube-api-access-g7xm5\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.048277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.048242    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:20.332943 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:22:20.332910    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f1534e_4438_4789_aeee_f944b3dd15fc.slice/crio-ccaea8c28c3e51a90cbca1470340c936c327b56d1d7d8638075a299903aac990 WatchSource:0}: Error finding container ccaea8c28c3e51a90cbca1470340c936c327b56d1d7d8638075a299903aac990: Status 404 returned error can't find the container with id ccaea8c28c3e51a90cbca1470340c936c327b56d1d7d8638075a299903aac990
Oct 30 19:22:20.333083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.332955    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"]
Oct 30 19:22:20.335447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.335418    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:20.335527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.335483    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:20.335527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.335514    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:20.833014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.832976    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" event={"ID":"77f1534e-4438-4789-aeee-f944b3dd15fc","Type":"ContainerStarted","Data":"c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5"}
Oct 30 19:22:20.833446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.833040    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" event={"ID":"77f1534e-4438-4789-aeee-f944b3dd15fc","Type":"ContainerStarted","Data":"ccaea8c28c3e51a90cbca1470340c936c327b56d1d7d8638075a299903aac990"}
Oct 30 19:22:20.859697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:20.859649    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podStartSLOduration=1.859633154 podStartE2EDuration="1.859633154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:20.856676363 +0000 UTC m=+922.853591750" watchObservedRunningTime="2026-10-30 19:22:20.859633154 +0000 UTC m=+922.856548538"
Oct 30 19:22:21.048803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:21.048773    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:21.050293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:21.050266    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:22:21.050423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:21.050316    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:22:22.060083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:22.056454    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:22:22.060083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:22.056530    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:22:23.069086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:23.066471    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:22:23.069086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:23.066542    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:22:24.073658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:24.073612    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:22:24.074154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:24.073685    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:22:25.056089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:25.051419    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:22:25.056089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:25.051479    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:22:26.057437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:26.057396    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:22:26.058004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:26.057973    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:22:27.049012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:27.048977    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:22:27.049241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:27.049042    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:22:28.053236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:28.053204    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:28.053687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:28.053668    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:28.054403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:28.054381    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:22:29.629668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.629631    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"]
Oct 30 19:22:29.637149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.636399    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.641451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.641421    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"]
Oct 30 19:22:29.641670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.641554    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:29.642563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.642536    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:22:29.674766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.674725    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c654b3-7nzlq\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.674921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.674779    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngbx\" (UniqueName: \"kubernetes.io/projected/adba99d3-c9f9-43b4-89ef-1a27db198915-kube-api-access-zngbx\") pod \"maas-api-revoke-sub-44c654b3-7nzlq\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.674921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.674858    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c654b3-7nzlq\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.776982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.776926    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c654b3-7nzlq\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.777400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.777034    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngbx\" (UniqueName: \"kubernetes.io/projected/adba99d3-c9f9-43b4-89ef-1a27db198915-kube-api-access-zngbx\") pod \"maas-api-revoke-sub-44c654b3-7nzlq\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.777400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.777110    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c654b3-7nzlq\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.778000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.777965    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c654b3-7nzlq\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.780152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.780125    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c654b3-7nzlq\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.789845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.789820    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngbx\" (UniqueName: \"kubernetes.io/projected/adba99d3-c9f9-43b4-89ef-1a27db198915-kube-api-access-zngbx\") pod \"maas-api-revoke-sub-44c654b3-7nzlq\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:29.954307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:29.954221    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:30.234023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:30.233997    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"]
Oct 30 19:22:30.235035 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:22:30.234999    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadba99d3_c9f9_43b4_89ef_1a27db198915.slice/crio-544a0e7f9c2cc7ec912f749c760d82dcf3cc8f1bfda3681da5a0ba173b4a3713 WatchSource:0}: Error finding container 544a0e7f9c2cc7ec912f749c760d82dcf3cc8f1bfda3681da5a0ba173b4a3713: Status 404 returned error can't find the container with id 544a0e7f9c2cc7ec912f749c760d82dcf3cc8f1bfda3681da5a0ba173b4a3713
Oct 30 19:22:30.923658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:30.923569    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq" event={"ID":"adba99d3-c9f9-43b4-89ef-1a27db198915","Type":"ContainerStarted","Data":"a0e0e3d121f35cc8d2b52d9c43fa37430c03a67cd13b29073d6ce337407145a9"}
Oct 30 19:22:30.923658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:30.923633    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq" event={"ID":"adba99d3-c9f9-43b4-89ef-1a27db198915","Type":"ContainerStarted","Data":"544a0e7f9c2cc7ec912f749c760d82dcf3cc8f1bfda3681da5a0ba173b4a3713"}
Oct 30 19:22:30.944499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:30.944449    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq" podStartSLOduration=1.9444362179999999 podStartE2EDuration="1.944436218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:30.942178891 +0000 UTC m=+932.939094298" watchObservedRunningTime="2026-10-30 19:22:30.944436218 +0000 UTC m=+932.941351601"
Oct 30 19:22:31.931953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:31.931913    6777 generic.go:356] "Generic (PLEG): container finished" podID="adba99d3-c9f9-43b4-89ef-1a27db198915" containerID="a0e0e3d121f35cc8d2b52d9c43fa37430c03a67cd13b29073d6ce337407145a9" exitCode=0
Oct 30 19:22:31.932368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:31.931966    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq" event={"ID":"adba99d3-c9f9-43b4-89ef-1a27db198915","Type":"ContainerDied","Data":"a0e0e3d121f35cc8d2b52d9c43fa37430c03a67cd13b29073d6ce337407145a9"}
Oct 30 19:22:33.105233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.105204    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:33.234784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.234704    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-openshift-ca\") pod \"adba99d3-c9f9-43b4-89ef-1a27db198915\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") "
Oct 30 19:22:33.234784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.234758    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adba99d3-c9f9-43b4-89ef-1a27db198915-kube-api-access-zngbx\" (UniqueName: \"kubernetes.io/projected/adba99d3-c9f9-43b4-89ef-1a27db198915-kube-api-access-zngbx\") pod \"adba99d3-c9f9-43b4-89ef-1a27db198915\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") "
Oct 30 19:22:33.235013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.234814    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-xks-ca\") pod \"adba99d3-c9f9-43b4-89ef-1a27db198915\" (UID: \"adba99d3-c9f9-43b4-89ef-1a27db198915\") "
Oct 30 19:22:33.235205 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.235175    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-openshift-ca" pod "adba99d3-c9f9-43b4-89ef-1a27db198915" (UID: "adba99d3-c9f9-43b4-89ef-1a27db198915"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:33.235689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.235647    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:33.237091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.237052    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adba99d3-c9f9-43b4-89ef-1a27db198915-kube-api-access-zngbx" pod "adba99d3-c9f9-43b4-89ef-1a27db198915" (UID: "adba99d3-c9f9-43b4-89ef-1a27db198915"). InnerVolumeSpecName "kube-api-access-zngbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:33.237438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.237425    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-xks-ca" pod "adba99d3-c9f9-43b4-89ef-1a27db198915" (UID: "adba99d3-c9f9-43b4-89ef-1a27db198915"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:33.337045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.337010    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngbx\" (UniqueName: \"kubernetes.io/projected/adba99d3-c9f9-43b4-89ef-1a27db198915-kube-api-access-zngbx\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:33.337045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:33.337045    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adba99d3-c9f9-43b4-89ef-1a27db198915-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:34.008534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:34.004337    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq" event={"ID":"adba99d3-c9f9-43b4-89ef-1a27db198915","Type":"ContainerDied","Data":"544a0e7f9c2cc7ec912f749c760d82dcf3cc8f1bfda3681da5a0ba173b4a3713"}
Oct 30 19:22:34.008534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:34.004422    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="544a0e7f9c2cc7ec912f749c760d82dcf3cc8f1bfda3681da5a0ba173b4a3713"
Oct 30 19:22:34.008534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:34.004505    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"
Oct 30 19:22:38.545546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.545512    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"]
Oct 30 19:22:38.546155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.546138    6777 cpu_manager.go:414] "containerMap: removing container" podUID="adba99d3-c9f9-43b4-89ef-1a27db198915" containerName="revoke-keys"
Oct 30 19:22:38.546155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.546155    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adba99d3-c9f9-43b4-89ef-1a27db198915" containerName="revoke-keys"
Oct 30 19:22:38.546302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.546293    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="adba99d3-c9f9-43b4-89ef-1a27db198915" containerName="revoke-keys"
Oct 30 19:22:38.549746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.549725    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.554427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.554401    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:38.555319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.555297    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:22:38.557167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.557144    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"]
Oct 30 19:22:38.671868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.671827    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bcee05-fjdfj\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.672041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.671885    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bcee05-fjdfj\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.672173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.672147    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pln5\" (UniqueName: \"kubernetes.io/projected/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-kube-api-access-2pln5\") pod \"maas-api-revoke-sub-c0bcee05-fjdfj\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.773035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.773002    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pln5\" (UniqueName: \"kubernetes.io/projected/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-kube-api-access-2pln5\") pod \"maas-api-revoke-sub-c0bcee05-fjdfj\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.773239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.773158    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bcee05-fjdfj\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.773239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.773194    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bcee05-fjdfj\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.773798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.773772    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bcee05-fjdfj\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.775609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.775585    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bcee05-fjdfj\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.785896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.785870    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pln5\" (UniqueName: \"kubernetes.io/projected/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-kube-api-access-2pln5\") pod \"maas-api-revoke-sub-c0bcee05-fjdfj\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:38.865779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:38.865707    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:39.146383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:39.146250    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"]
Oct 30 19:22:39.146621 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:22:39.146594    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c24133_5d0f_43ac_a702_6fd1fc69f5ae.slice/crio-a798224c6a01d0597e2b2c0c99319d51d384dafdb1a68cb5df3f0e37394719ba WatchSource:0}: Error finding container a798224c6a01d0597e2b2c0c99319d51d384dafdb1a68cb5df3f0e37394719ba: Status 404 returned error can't find the container with id a798224c6a01d0597e2b2c0c99319d51d384dafdb1a68cb5df3f0e37394719ba
Oct 30 19:22:40.086855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:40.086824    6777 generic.go:356] "Generic (PLEG): container finished" podID="50c24133-5d0f-43ac-a702-6fd1fc69f5ae" containerID="0c87708395f9fa838cf00d89f1bf392f172e038741bd8f3e11a58fc3ae70ee3f" exitCode=0
Oct 30 19:22:40.087261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:40.086915    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj" event={"ID":"50c24133-5d0f-43ac-a702-6fd1fc69f5ae","Type":"ContainerDied","Data":"0c87708395f9fa838cf00d89f1bf392f172e038741bd8f3e11a58fc3ae70ee3f"}
Oct 30 19:22:40.087261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:40.086997    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj" event={"ID":"50c24133-5d0f-43ac-a702-6fd1fc69f5ae","Type":"ContainerStarted","Data":"a798224c6a01d0597e2b2c0c99319d51d384dafdb1a68cb5df3f0e37394719ba"}
Oct 30 19:22:41.298471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.298442    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:41.311567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.311539    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-kube-api-access-2pln5\" (UniqueName: \"kubernetes.io/projected/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-kube-api-access-2pln5\") pod \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") "
Oct 30 19:22:41.311712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.311580    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-openshift-ca\") pod \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") "
Oct 30 19:22:41.311712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.311606    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-xks-ca\") pod \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\" (UID: \"50c24133-5d0f-43ac-a702-6fd1fc69f5ae\") "
Oct 30 19:22:41.312028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.312007    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-openshift-ca" pod "50c24133-5d0f-43ac-a702-6fd1fc69f5ae" (UID: "50c24133-5d0f-43ac-a702-6fd1fc69f5ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:41.312321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.312300    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:41.313753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.313722    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-xks-ca" pod "50c24133-5d0f-43ac-a702-6fd1fc69f5ae" (UID: "50c24133-5d0f-43ac-a702-6fd1fc69f5ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:41.313851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.313764    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-kube-api-access-2pln5" pod "50c24133-5d0f-43ac-a702-6fd1fc69f5ae" (UID: "50c24133-5d0f-43ac-a702-6fd1fc69f5ae"). InnerVolumeSpecName "kube-api-access-2pln5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:41.413765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.413664    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pln5\" (UniqueName: \"kubernetes.io/projected/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-kube-api-access-2pln5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:41.413765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:41.413703    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c24133-5d0f-43ac-a702-6fd1fc69f5ae-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:42.101187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.101157    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"
Oct 30 19:22:42.101339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.101188    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj" event={"ID":"50c24133-5d0f-43ac-a702-6fd1fc69f5ae","Type":"ContainerDied","Data":"a798224c6a01d0597e2b2c0c99319d51d384dafdb1a68cb5df3f0e37394719ba"}
Oct 30 19:22:42.101339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.101235    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a798224c6a01d0597e2b2c0c99319d51d384dafdb1a68cb5df3f0e37394719ba"
Oct 30 19:22:42.282942 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.282905    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:22:42.754757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.754724    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"]
Oct 30 19:22:42.755463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.755446    6777 cpu_manager.go:414] "containerMap: removing container" podUID="50c24133-5d0f-43ac-a702-6fd1fc69f5ae" containerName="revoke-keys"
Oct 30 19:22:42.755512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.755465    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c24133-5d0f-43ac-a702-6fd1fc69f5ae" containerName="revoke-keys"
Oct 30 19:22:42.755587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.755577    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c24133-5d0f-43ac-a702-6fd1fc69f5ae" containerName="revoke-keys"
Oct 30 19:22:42.759824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.759797    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:42.774097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.774048    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:22:42.775805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.775752    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:42.783359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.783323    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"]
Oct 30 19:22:42.833763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.833726    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzzc\" (UniqueName: \"kubernetes.io/projected/9d2644a8-ce28-4367-830d-6e0bc869a22f-kube-api-access-zmzzc\") pod \"maas-api-revoke-sub-254077ba-frb4v\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:42.834154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.834133    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-254077ba-frb4v\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:42.834282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.834264    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-254077ba-frb4v\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:42.936320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.936032    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-254077ba-frb4v\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:42.936320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.936087    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-254077ba-frb4v\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:42.936320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.936239    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzzc\" (UniqueName: \"kubernetes.io/projected/9d2644a8-ce28-4367-830d-6e0bc869a22f-kube-api-access-zmzzc\") pod \"maas-api-revoke-sub-254077ba-frb4v\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:42.936907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.936880    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-254077ba-frb4v\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:42.938546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.938521    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-254077ba-frb4v\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:42.949113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:42.949089    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzzc\" (UniqueName: \"kubernetes.io/projected/9d2644a8-ce28-4367-830d-6e0bc869a22f-kube-api-access-zmzzc\") pod \"maas-api-revoke-sub-254077ba-frb4v\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:43.133019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.132989    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:43.467698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.467646    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"]
Oct 30 19:22:43.687324 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:22:43.687278    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:22:43.691972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.691939    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"]
Oct 30 19:22:43.785415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.785385    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"]
Oct 30 19:22:43.803082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.786033    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:43.819312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.819273    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-642gd\""
Oct 30 19:22:43.819507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.819356    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:22:43.819507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.819404    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:22:43.819507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.819443    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:22:43.994206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.992671    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:43.994206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.992726    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:43.994206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.992781    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:43.994206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.992868    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:43.994206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.992941    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwfn2\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-kube-api-access-dwfn2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:43.994206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.993698    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:43.994679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:43.994305    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.103082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.102082    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.103082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.102176    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.108081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.103604    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.108081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.103825    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.108081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.103964    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.147488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.141792    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.147488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.126262    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.147488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.144123    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwfn2\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-kube-api-access-dwfn2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.147488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.147432    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.152078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.144861    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.152078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.150484    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.189096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.183294    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.208272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.208199    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwfn2\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-kube-api-access-dwfn2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.208597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.208488    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v" event={"ID":"9d2644a8-ce28-4367-830d-6e0bc869a22f","Type":"ContainerStarted","Data":"309613fc5b47d0a01804fac6c9ec8150b1b37d863dd3cca7ce3c312779a2d5ad"}
Oct 30 19:22:44.208597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.208555    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v" event={"ID":"9d2644a8-ce28-4367-830d-6e0bc869a22f","Type":"ContainerStarted","Data":"8ae5ededafffe928b40c9bcc82fa8050fcc8c70c0bd0f6c14efb6953fa974d73"}
Oct 30 19:22:44.223088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.221710    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk2r9\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:44.262920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.262822    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v" podStartSLOduration=2.262800602 podStartE2EDuration="2.262800602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:44.262257074 +0000 UTC m=+946.259172459" watchObservedRunningTime="2026-10-30 19:22:44.262800602 +0000 UTC m=+946.259715986"
Oct 30 19:22:44.316566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:44.316414    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:45.265100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.172930    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:22:45.265100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.198656    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.300294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.300252    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.331824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.331174    6777 generic.go:356] "Generic (PLEG): container finished" podID="9d2644a8-ce28-4367-830d-6e0bc869a22f" containerID="309613fc5b47d0a01804fac6c9ec8150b1b37d863dd3cca7ce3c312779a2d5ad" exitCode=0
Oct 30 19:22:45.331824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.331381    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v" event={"ID":"9d2644a8-ce28-4367-830d-6e0bc869a22f","Type":"ContainerDied","Data":"309613fc5b47d0a01804fac6c9ec8150b1b37d863dd3cca7ce3c312779a2d5ad"}
Oct 30 19:22:45.399263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.399225    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:22:45.399568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.399537    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.474115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.469336    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.640809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.640759    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.654900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.654865    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"]
Oct 30 19:22:45.976285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.976171    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.982218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:45.982171    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:46.041693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.011159    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:46.043675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.039171    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:46.194332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.190679    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:46.200230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.200188    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:46.222498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.204041    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:22:46.222498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.204126    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:22:46.271334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.271246    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:22:46.300728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.300683    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:46.320590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.320558    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:22:46.421114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.417950    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:22:46.421114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.417991    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9" event={"ID":"61950875-cf6e-4bea-8e3c-ef6a6c6c4499","Type":"ContainerStarted","Data":"52820a57af8db3b0f4252b2d38727a055b954949e7eceb246048afeafe85aea5"}
Oct 30 19:22:46.514122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.514088    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:22:46.522153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.521316    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:46.522153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.521994    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:22:46.692924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.689820    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:22:46.927033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.926999    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:46.955826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.955791    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2644a8-ce28-4367-830d-6e0bc869a22f-kube-api-access-zmzzc\" (UniqueName: \"kubernetes.io/projected/9d2644a8-ce28-4367-830d-6e0bc869a22f-kube-api-access-zmzzc\") pod \"9d2644a8-ce28-4367-830d-6e0bc869a22f\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") "
Oct 30 19:22:46.956010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.955908    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-openshift-ca\") pod \"9d2644a8-ce28-4367-830d-6e0bc869a22f\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") "
Oct 30 19:22:46.956010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.955964    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-xks-ca\") pod \"9d2644a8-ce28-4367-830d-6e0bc869a22f\" (UID: \"9d2644a8-ce28-4367-830d-6e0bc869a22f\") "
Oct 30 19:22:46.956449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.956423    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-openshift-ca" pod "9d2644a8-ce28-4367-830d-6e0bc869a22f" (UID: "9d2644a8-ce28-4367-830d-6e0bc869a22f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:46.956694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.956676    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.958499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.958475    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2644a8-ce28-4367-830d-6e0bc869a22f-kube-api-access-zmzzc" pod "9d2644a8-ce28-4367-830d-6e0bc869a22f" (UID: "9d2644a8-ce28-4367-830d-6e0bc869a22f"). InnerVolumeSpecName "kube-api-access-zmzzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:46.963989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:46.963959    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-xks-ca" pod "9d2644a8-ce28-4367-830d-6e0bc869a22f" (UID: "9d2644a8-ce28-4367-830d-6e0bc869a22f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:47.058419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:47.058341    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmzzc\" (UniqueName: \"kubernetes.io/projected/9d2644a8-ce28-4367-830d-6e0bc869a22f-kube-api-access-zmzzc\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.058419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:47.058368    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2644a8-ce28-4367-830d-6e0bc869a22f-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.417131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:47.416558    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"
Oct 30 19:22:47.417131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:47.416664    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v" event={"ID":"9d2644a8-ce28-4367-830d-6e0bc869a22f","Type":"ContainerDied","Data":"8ae5ededafffe928b40c9bcc82fa8050fcc8c70c0bd0f6c14efb6953fa974d73"}
Oct 30 19:22:47.417131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:47.416716    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ae5ededafffe928b40c9bcc82fa8050fcc8c70c0bd0f6c14efb6953fa974d73"
Oct 30 19:22:49.081954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:49.081921    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:49.082262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:49.081985    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:22:49.433519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:49.433439    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9" event={"ID":"61950875-cf6e-4bea-8e3c-ef6a6c6c4499","Type":"ContainerStarted","Data":"45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605"}
Oct 30 19:22:49.433673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:49.433546    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:49.462623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:49.462561    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9" podStartSLOduration=3.077356748 podStartE2EDuration="6.462545924s" totalImagesPullingTime="3.385189176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:49.460665079 +0000 UTC m=+951.457580462" watchObservedRunningTime="2026-10-30 19:22:49.462545924 +0000 UTC m=+951.459461307"
Oct 30 19:22:50.444433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:50.444405    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:22:55.096153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.096115    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"]
Oct 30 19:22:55.096730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.096716    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9d2644a8-ce28-4367-830d-6e0bc869a22f" containerName="revoke-keys"
Oct 30 19:22:55.096785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.096731    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2644a8-ce28-4367-830d-6e0bc869a22f" containerName="revoke-keys"
Oct 30 19:22:55.096864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.096855    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2644a8-ce28-4367-830d-6e0bc869a22f" containerName="revoke-keys"
Oct 30 19:22:55.100863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.100841    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.105859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.105836    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:22:55.105986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.105926    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:55.108796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.108767    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"]
Oct 30 19:22:55.201289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.201250    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3734c02f-d420-48f8-8029-75540531648b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ae66ae-lf49h\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.201460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.201335    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3734c02f-d420-48f8-8029-75540531648b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ae66ae-lf49h\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.201460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.201445    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9rg\" (UniqueName: \"kubernetes.io/projected/3734c02f-d420-48f8-8029-75540531648b-kube-api-access-bv9rg\") pod \"maas-api-revoke-sub-06ae66ae-lf49h\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.303401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.303368    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3734c02f-d420-48f8-8029-75540531648b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ae66ae-lf49h\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.303604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.303466    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9rg\" (UniqueName: \"kubernetes.io/projected/3734c02f-d420-48f8-8029-75540531648b-kube-api-access-bv9rg\") pod \"maas-api-revoke-sub-06ae66ae-lf49h\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.303652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.303646    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3734c02f-d420-48f8-8029-75540531648b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ae66ae-lf49h\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.304328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.304299    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3734c02f-d420-48f8-8029-75540531648b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ae66ae-lf49h\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.305931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.305909    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3734c02f-d420-48f8-8029-75540531648b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ae66ae-lf49h\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.316516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.316492    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9rg\" (UniqueName: \"kubernetes.io/projected/3734c02f-d420-48f8-8029-75540531648b-kube-api-access-bv9rg\") pod \"maas-api-revoke-sub-06ae66ae-lf49h\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.416863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.416773    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:55.699939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:55.699908    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"]
Oct 30 19:22:55.700762 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:22:55.700737    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3734c02f_d420_48f8_8029_75540531648b.slice/crio-1beda8a9e586f67d1b54a2d8fbce8a714689efb9dd1d79c929a0ce0bc5a6306d WatchSource:0}: Error finding container 1beda8a9e586f67d1b54a2d8fbce8a714689efb9dd1d79c929a0ce0bc5a6306d: Status 404 returned error can't find the container with id 1beda8a9e586f67d1b54a2d8fbce8a714689efb9dd1d79c929a0ce0bc5a6306d
Oct 30 19:22:56.505130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:56.505086    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h" event={"ID":"3734c02f-d420-48f8-8029-75540531648b","Type":"ContainerStarted","Data":"fbdaa248cecb111164ed799b3da4890a8925f2b654cd069e4a46707d3e1bdb3d"}
Oct 30 19:22:56.505680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:56.505656    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h" event={"ID":"3734c02f-d420-48f8-8029-75540531648b","Type":"ContainerStarted","Data":"1beda8a9e586f67d1b54a2d8fbce8a714689efb9dd1d79c929a0ce0bc5a6306d"}
Oct 30 19:22:56.525579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:56.525525    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h" podStartSLOduration=1.525512341 podStartE2EDuration="1.525512341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:56.524662878 +0000 UTC m=+958.521578267" watchObservedRunningTime="2026-10-30 19:22:56.525512341 +0000 UTC m=+958.522427724"
Oct 30 19:22:57.513017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:57.512979    6777 generic.go:356] "Generic (PLEG): container finished" podID="3734c02f-d420-48f8-8029-75540531648b" containerID="fbdaa248cecb111164ed799b3da4890a8925f2b654cd069e4a46707d3e1bdb3d" exitCode=0
Oct 30 19:22:57.513425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:57.513026    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h" event={"ID":"3734c02f-d420-48f8-8029-75540531648b","Type":"ContainerDied","Data":"fbdaa248cecb111164ed799b3da4890a8925f2b654cd069e4a46707d3e1bdb3d"}
Oct 30 19:22:58.682844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.682819    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:22:58.854573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.854543    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3734c02f-d420-48f8-8029-75540531648b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3734c02f-d420-48f8-8029-75540531648b-maas-api-openshift-ca\") pod \"3734c02f-d420-48f8-8029-75540531648b\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") "
Oct 30 19:22:58.854786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.854606    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3734c02f-d420-48f8-8029-75540531648b-kube-api-access-bv9rg\" (UniqueName: \"kubernetes.io/projected/3734c02f-d420-48f8-8029-75540531648b-kube-api-access-bv9rg\") pod \"3734c02f-d420-48f8-8029-75540531648b\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") "
Oct 30 19:22:58.854850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.854801    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3734c02f-d420-48f8-8029-75540531648b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3734c02f-d420-48f8-8029-75540531648b-maas-api-xks-ca\") pod \"3734c02f-d420-48f8-8029-75540531648b\" (UID: \"3734c02f-d420-48f8-8029-75540531648b\") "
Oct 30 19:22:58.854902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.854840    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3734c02f-d420-48f8-8029-75540531648b-maas-api-openshift-ca" pod "3734c02f-d420-48f8-8029-75540531648b" (UID: "3734c02f-d420-48f8-8029-75540531648b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:58.855381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.855357    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3734c02f-d420-48f8-8029-75540531648b-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:58.856797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.856767    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3734c02f-d420-48f8-8029-75540531648b-kube-api-access-bv9rg" pod "3734c02f-d420-48f8-8029-75540531648b" (UID: "3734c02f-d420-48f8-8029-75540531648b"). InnerVolumeSpecName "kube-api-access-bv9rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:58.856913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.856889    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3734c02f-d420-48f8-8029-75540531648b-maas-api-xks-ca" pod "3734c02f-d420-48f8-8029-75540531648b" (UID: "3734c02f-d420-48f8-8029-75540531648b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:58.956993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.956961    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3734c02f-d420-48f8-8029-75540531648b-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:58.956993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:58.956992    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv9rg\" (UniqueName: \"kubernetes.io/projected/3734c02f-d420-48f8-8029-75540531648b-kube-api-access-bv9rg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:59.547448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:59.547401    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h" event={"ID":"3734c02f-d420-48f8-8029-75540531648b","Type":"ContainerDied","Data":"1beda8a9e586f67d1b54a2d8fbce8a714689efb9dd1d79c929a0ce0bc5a6306d"}
Oct 30 19:22:59.547760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:59.547740    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1beda8a9e586f67d1b54a2d8fbce8a714689efb9dd1d79c929a0ce0bc5a6306d"
Oct 30 19:22:59.547944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:22:59.547929    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"
Oct 30 19:23:01.984083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:01.983127    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:23:02.054773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:02.054184    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:23:02.101094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:02.099223    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:05.301400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.301360    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"]
Oct 30 19:23:05.302282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.302256    6777 cpu_manager.go:414] "containerMap: removing container" podUID="3734c02f-d420-48f8-8029-75540531648b" containerName="revoke-keys"
Oct 30 19:23:05.302282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.302283    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3734c02f-d420-48f8-8029-75540531648b" containerName="revoke-keys"
Oct 30 19:23:05.302498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.302484    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="3734c02f-d420-48f8-8029-75540531648b" containerName="revoke-keys"
Oct 30 19:23:05.306907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.306864    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.334041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.334002    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"]
Oct 30 19:23:05.340174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.340143    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"]
Oct 30 19:23:05.346175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.346148    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.349148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.349120    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"]
Oct 30 19:23:05.350758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.350731    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.350899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.350766    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.351073 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.351035    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f965r\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-kube-api-access-f965r\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.351153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.351122    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.351213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.351195    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.351251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.351226    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.351289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.351261    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.351360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.351344    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.351459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.351445    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.430241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.430204    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"]
Oct 30 19:23:05.434423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.434401    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.453294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453264    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.453463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453306    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.453463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453445    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.453577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453521    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f965r\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-kube-api-access-f965r\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.453632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453604    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.453679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453650    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4959594f-0d2e-49ed-a21a-17db03609138-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.453725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453687    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.453725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453707    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.453822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453737    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.453822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453763    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7h6\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-kube-api-access-hk7h6\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.453923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453877    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4959594f-0d2e-49ed-a21a-17db03609138-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.453923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453908    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.454032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.453951    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.454032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.454013    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.454198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.454048    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.454198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.454154    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.454304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.454195    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.454304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.454257    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.454407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.454300    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.455381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.454652    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.455381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.454947    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.455381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.455235    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.455381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.455245    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.456574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.456543    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.457841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.457813    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.460500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.460481    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"]
Oct 30 19:23:05.473687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.473660    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f965r\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-kube-api-access-f965r\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.474693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.474673    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.555907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.555821    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.555907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.555884    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.556118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.555960    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4959594f-0d2e-49ed-a21a-17db03609138-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.556118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556068    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.556188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556135    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7h6\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-kube-api-access-hk7h6\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.556188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556156    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.556260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556218    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4959594f-0d2e-49ed-a21a-17db03609138-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.556302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556257    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.556302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556288    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7489c99-67e2-438e-9593-3f552755ff5c-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.556382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556337    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.556425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556386    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.556473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556424    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.556566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556484    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.556566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556556    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.556666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556630    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7489c99-67e2-438e-9593-3f552755ff5c-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.556721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556663    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7z47\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-kube-api-access-w7z47\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.556721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556694    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.556721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556702    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.556905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.556885    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.557029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.557011    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4959594f-0d2e-49ed-a21a-17db03609138-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.557134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.557069    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.557134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.557098    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.557373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.557349    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.558534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.558513    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.558611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.558578    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4959594f-0d2e-49ed-a21a-17db03609138-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.571591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.571554    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.571908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.571889    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7h6\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-kube-api-access-hk7h6\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.637973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.637938    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:05.660260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.660216    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7489c99-67e2-438e-9593-3f552755ff5c-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.660402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.660275    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7z47\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-kube-api-access-w7z47\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.660402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.660306    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.660514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.660419    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.660514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.660472    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.660605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.660514    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.660605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.660549    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.660707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.660645    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7489c99-67e2-438e-9593-3f552755ff5c-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.660779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.660763    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.661074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.661037    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7489c99-67e2-438e-9593-3f552755ff5c-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.661198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.661129    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.661281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.661255    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.661354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.661334    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.661534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.661512    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.662973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.662939    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:05.663420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.663396    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.663924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.663900    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7489c99-67e2-438e-9593-3f552755ff5c-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.674099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.674070    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7z47\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-kube-api-access-w7z47\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.680543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.680506    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.748377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.748033    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:05.938724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.938690    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"]
Oct 30 19:23:05.939452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.939148    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:05.939452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.939280    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:05.939452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.939380    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:05.972140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.972111    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"]
Oct 30 19:23:05.979424 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:23:05.979395    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4959594f_0d2e_49ed_a21a_17db03609138.slice/crio-062b4a9605dc6727d9df2d881f844db24ddbb6723e1785a355b1fa409d2fdd5d WatchSource:0}: Error finding container 062b4a9605dc6727d9df2d881f844db24ddbb6723e1785a355b1fa409d2fdd5d: Status 404 returned error can't find the container with id 062b4a9605dc6727d9df2d881f844db24ddbb6723e1785a355b1fa409d2fdd5d
Oct 30 19:23:05.983045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.982992    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:05.983473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.983116    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:05.983473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:05.983156    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:06.047724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.047696    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"]
Oct 30 19:23:06.049499 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:23:06.049474    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7489c99_67e2_438e_9593_3f552755ff5c.slice/crio-f2119458459d0cb42eb3463c16920a4a5f4b63b4a71bc35a5e9c1133cde2a9bd WatchSource:0}: Error finding container f2119458459d0cb42eb3463c16920a4a5f4b63b4a71bc35a5e9c1133cde2a9bd: Status 404 returned error can't find the container with id f2119458459d0cb42eb3463c16920a4a5f4b63b4a71bc35a5e9c1133cde2a9bd
Oct 30 19:23:06.052126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.052093    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:06.052232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.052162    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:06.052232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.052191    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:06.647357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.647193    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" event={"ID":"e7489c99-67e2-438e-9593-3f552755ff5c","Type":"ContainerStarted","Data":"24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee"}
Oct 30 19:23:06.648275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.648209    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" event={"ID":"e7489c99-67e2-438e-9593-3f552755ff5c","Type":"ContainerStarted","Data":"f2119458459d0cb42eb3463c16920a4a5f4b63b4a71bc35a5e9c1133cde2a9bd"}
Oct 30 19:23:06.652507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.652181    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" event={"ID":"4959594f-0d2e-49ed-a21a-17db03609138","Type":"ContainerStarted","Data":"e48ef53d1197e7fc4b2dbcfdfde524b3e3cc476a4ab43a55354b4b674c9ade9a"}
Oct 30 19:23:06.652507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.652250    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" event={"ID":"4959594f-0d2e-49ed-a21a-17db03609138","Type":"ContainerStarted","Data":"062b4a9605dc6727d9df2d881f844db24ddbb6723e1785a355b1fa409d2fdd5d"}
Oct 30 19:23:06.659122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.654188    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" event={"ID":"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3","Type":"ContainerStarted","Data":"ce1dfb49eff11602c1ce71724b00d87b36314ea43145b0de31b63324bd78dc4b"}
Oct 30 19:23:06.659122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.654261    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" event={"ID":"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3","Type":"ContainerStarted","Data":"986f7fd8515f6bce29c3b714333573d1d98fc9f9b7d1f1824b0f04e149660357"}
Oct 30 19:23:06.663746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.663714    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:06.692295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.692212    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" podStartSLOduration=1.692193815 podStartE2EDuration="1.692193815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:06.675102384 +0000 UTC m=+968.672017767" watchObservedRunningTime="2026-10-30 19:23:06.692193815 +0000 UTC m=+968.689109202"
Oct 30 19:23:06.731099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.730974    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" podStartSLOduration=1.730955227 podStartE2EDuration="1.730955227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:06.717424686 +0000 UTC m=+968.714340096" watchObservedRunningTime="2026-10-30 19:23:06.730955227 +0000 UTC m=+968.727870625"
Oct 30 19:23:06.748519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.748477    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:06.757434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.757383    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" podStartSLOduration=1.757368707 podStartE2EDuration="1.757368707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:06.755343392 +0000 UTC m=+968.752258777" watchObservedRunningTime="2026-10-30 19:23:06.757368707 +0000 UTC m=+968.754284146"
Oct 30 19:23:06.830185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.830145    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"]
Oct 30 19:23:06.842432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.842400    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:06.846715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.846668    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"]
Oct 30 19:23:06.936634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.936545    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:07.021210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:06.968409    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:07.022454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.022421    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58a1d276-kt4v5\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.023297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.023272    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8shz\" (UniqueName: \"kubernetes.io/projected/1d0693cb-88af-47d6-adf3-083d4466a04c-kube-api-access-n8shz\") pod \"maas-api-revoke-sub-58a1d276-kt4v5\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.023531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.023509    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58a1d276-kt4v5\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.138950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.138912    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58a1d276-kt4v5\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.139586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.139558    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8shz\" (UniqueName: \"kubernetes.io/projected/1d0693cb-88af-47d6-adf3-083d4466a04c-kube-api-access-n8shz\") pod \"maas-api-revoke-sub-58a1d276-kt4v5\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.140603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.140256    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58a1d276-kt4v5\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.140603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.140112    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58a1d276-kt4v5\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.142865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.142839    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58a1d276-kt4v5\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.182560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.182523    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8shz\" (UniqueName: \"kubernetes.io/projected/1d0693cb-88af-47d6-adf3-083d4466a04c-kube-api-access-n8shz\") pod \"maas-api-revoke-sub-58a1d276-kt4v5\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.441737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.441699    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:07.638812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.638774    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:07.639854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.639830    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:07.639997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.639875    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 19:23:07.732377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.732291    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:07.732377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.732347    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 19:23:07.853235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.853194    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:23:07.863087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:07.853522    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:23:08.430809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:08.430306    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"]
Oct 30 19:23:08.639162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:08.639124    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:08.639337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:08.639185    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 19:23:08.684895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:08.684799    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:08.685175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:08.685144    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 19:23:08.740111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:08.738732    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5" event={"ID":"1d0693cb-88af-47d6-adf3-083d4466a04c","Type":"ContainerStarted","Data":"87dfbdfabff36dbaef9c0dc55d4b06814c78adcb12b4cf3cf3b7f3ce823e0a91"}
Oct 30 19:23:08.757225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:08.757188    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:23:08.775431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:08.759002    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:23:09.638356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:09.638319    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:09.639086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:09.638377    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 19:23:09.687081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:09.678302    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:09.687081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:09.678362    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 19:23:09.755090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:09.748105    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5" event={"ID":"1d0693cb-88af-47d6-adf3-083d4466a04c","Type":"ContainerStarted","Data":"cf2de65edaf38017860f79eb9bce6cbf6ac94fe02e60e333c7d61a7f9aae1a81"}
Oct 30 19:23:09.798159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:09.785074    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:23:09.798159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:09.785151    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:23:09.831105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:09.824089    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5" podStartSLOduration=3.824043815 podStartE2EDuration="3.824043815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:09.800952143 +0000 UTC m=+971.797867533" watchObservedRunningTime="2026-10-30 19:23:09.824043815 +0000 UTC m=+971.820959201"
Oct 30 19:23:10.639283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:10.639245    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:10.639470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:10.639297    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 19:23:10.678083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:10.667455    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:10.678083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:10.667519    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 19:23:10.750091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:10.748383    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:23:10.750091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:10.748445    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:23:10.800083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:10.792024    6777 generic.go:356] "Generic (PLEG): container finished" podID="1d0693cb-88af-47d6-adf3-083d4466a04c" containerID="cf2de65edaf38017860f79eb9bce6cbf6ac94fe02e60e333c7d61a7f9aae1a81" exitCode=0
Oct 30 19:23:10.800083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:10.792170    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5" event={"ID":"1d0693cb-88af-47d6-adf3-083d4466a04c","Type":"ContainerDied","Data":"cf2de65edaf38017860f79eb9bce6cbf6ac94fe02e60e333c7d61a7f9aae1a81"}
Oct 30 19:23:11.638397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:11.638277    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:11.638397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:11.638327    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 19:23:11.683085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:11.682351    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:11.683085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:11.682413    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 19:23:11.756092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:11.753176    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:23:11.756092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:11.753240    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:23:12.444703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.444675    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:12.532612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.532562    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-xks-ca\") pod \"1d0693cb-88af-47d6-adf3-083d4466a04c\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") "
Oct 30 19:23:12.532799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.532723    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0693cb-88af-47d6-adf3-083d4466a04c-kube-api-access-n8shz\" (UniqueName: \"kubernetes.io/projected/1d0693cb-88af-47d6-adf3-083d4466a04c-kube-api-access-n8shz\") pod \"1d0693cb-88af-47d6-adf3-083d4466a04c\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") "
Oct 30 19:23:12.532851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.532828    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-openshift-ca\") pod \"1d0693cb-88af-47d6-adf3-083d4466a04c\" (UID: \"1d0693cb-88af-47d6-adf3-083d4466a04c\") "
Oct 30 19:23:12.533210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.533182    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-openshift-ca" pod "1d0693cb-88af-47d6-adf3-083d4466a04c" (UID: "1d0693cb-88af-47d6-adf3-083d4466a04c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:12.533505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.533482    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:12.535218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.535192    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-xks-ca" pod "1d0693cb-88af-47d6-adf3-083d4466a04c" (UID: "1d0693cb-88af-47d6-adf3-083d4466a04c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:12.535345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.535245    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0693cb-88af-47d6-adf3-083d4466a04c-kube-api-access-n8shz" pod "1d0693cb-88af-47d6-adf3-083d4466a04c" (UID: "1d0693cb-88af-47d6-adf3-083d4466a04c"). InnerVolumeSpecName "kube-api-access-n8shz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:12.634896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.634801    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8shz\" (UniqueName: \"kubernetes.io/projected/1d0693cb-88af-47d6-adf3-083d4466a04c-kube-api-access-n8shz\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:12.634896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.634847    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0693cb-88af-47d6-adf3-083d4466a04c-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:12.639137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.639100    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:12.639274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.639171    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 19:23:12.664253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.664201    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:12.664405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.664281    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 19:23:12.748884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.748844    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:23:12.749054 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.748955    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:23:12.851209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.851179    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"
Oct 30 19:23:12.851733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.851311    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5" event={"ID":"1d0693cb-88af-47d6-adf3-083d4466a04c","Type":"ContainerDied","Data":"87dfbdfabff36dbaef9c0dc55d4b06814c78adcb12b4cf3cf3b7f3ce823e0a91"}
Oct 30 19:23:12.851889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:12.851869    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87dfbdfabff36dbaef9c0dc55d4b06814c78adcb12b4cf3cf3b7f3ce823e0a91"
Oct 30 19:23:13.667084 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:23:13.666781    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:23:14.180751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:14.179125    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:14.180751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:14.179844    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:14.186262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:14.186225    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:23:14.236559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:14.234475    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:14.236559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:14.236123    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:14.241048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:14.237208    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:23:14.417513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:14.417478    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:14.417940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:14.417926    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:14.418775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:14.418745    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:23:20.208256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.208204    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"]
Oct 30 19:23:20.209103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.209081    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1d0693cb-88af-47d6-adf3-083d4466a04c" containerName="revoke-keys"
Oct 30 19:23:20.209216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.209104    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0693cb-88af-47d6-adf3-083d4466a04c" containerName="revoke-keys"
Oct 30 19:23:20.209301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.209288    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0693cb-88af-47d6-adf3-083d4466a04c" containerName="revoke-keys"
Oct 30 19:23:20.217741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.217708    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.229287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.229255    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"]
Oct 30 19:23:20.321723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.321685    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.321916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.321844    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.321916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.321873    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.322010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.321980    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.322053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.322014    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.322053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.322029    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.322216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.322192    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d4251f-32c1-421b-aa93-79794a66b2ac-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.322282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.322270    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.322316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.322303    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r992\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-kube-api-access-6r992\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.423988    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424034    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424080    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424161    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d4251f-32c1-421b-aa93-79794a66b2ac-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424214    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424256    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r992\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-kube-api-access-6r992\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424319    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424425    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424451    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424526    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424641    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424757    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424873    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.424980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.424959    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d4251f-32c1-421b-aa93-79794a66b2ac-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.426504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.426478    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.426655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.426640    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.436652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.436623    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.436750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.436716    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r992\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-kube-api-access-6r992\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.538850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.538821    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:20.826037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.826003    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"]
Oct 30 19:23:20.826178 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:23:20.826122    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d4251f_32c1_421b_aa93_79794a66b2ac.slice/crio-c1bd8ecc7e51bf9d77f43753c62d75aeb5b8fc6f8f0f91b80c14f0ceb78dfbd7 WatchSource:0}: Error finding container c1bd8ecc7e51bf9d77f43753c62d75aeb5b8fc6f8f0f91b80c14f0ceb78dfbd7: Status 404 returned error can't find the container with id c1bd8ecc7e51bf9d77f43753c62d75aeb5b8fc6f8f0f91b80c14f0ceb78dfbd7
Oct 30 19:23:20.828795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.828761    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:20.828905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.828842    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:20.828905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.828885    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:23:20.962585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.962545    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" event={"ID":"d5d4251f-32c1-421b-aa93-79794a66b2ac","Type":"ContainerStarted","Data":"3f192d20da9794a057741959b0f5ad644e7f2978395fb42912837efaff0590ab"}
Oct 30 19:23:20.962799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:20.962612    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" event={"ID":"d5d4251f-32c1-421b-aa93-79794a66b2ac","Type":"ContainerStarted","Data":"c1bd8ecc7e51bf9d77f43753c62d75aeb5b8fc6f8f0f91b80c14f0ceb78dfbd7"}
Oct 30 19:23:21.011863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:21.011804    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podStartSLOduration=1.011788116 podStartE2EDuration="1.011788116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:21.009285513 +0000 UTC m=+983.006200923" watchObservedRunningTime="2026-10-30 19:23:21.011788116 +0000 UTC m=+983.008703501"
Oct 30 19:23:21.539669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:21.539630    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:21.541623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:21.541592    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:21.541759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:21.541644    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:22.539194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:22.539156    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:22.539371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:22.539220    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:22.878712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:22.878627    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:23:23.540256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:23.540192    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:23.540475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:23.540281    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:24.544085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:24.540492    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:24.544085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:24.540570    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:25.053084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.047176    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.278084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.277166    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.299085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.287266    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:23:25.520160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.519223    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.543542 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.543497    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:25.543837 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.543798    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:25.615202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.615153    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.711098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.708168    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.962469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.962413    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:23:25.978694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:25.977108    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:23:26.055433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.011652    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.061083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.055855    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.061083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.056531    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:23:26.280532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.280490    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.319558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.319506    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.555150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.531150    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:23:26.626113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.620510    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:23:26.708443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.708396    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.868157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.866258    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:26.868157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.866349    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:26.909099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.906717    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:23:26.909099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.906809    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:23:26.909099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.906871    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:23:26.909099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.906927    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:23:26.909099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.906994    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:23:26.995083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.963186    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.995083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.971559    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:23:26.995083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.993904    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.021083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:26.995454    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:23:27.021083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.013420    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.021083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.013948    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:23:27.087097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.061223    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.340150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.281468    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.340150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.322226    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.537084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.533171    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.659896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.659781    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:23:27.688191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.659378    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 30 19:23:27.717359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.717321    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:27.717658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.717626    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:27.894552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.890766    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Oct 30 19:23:27.926807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.926706    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:23:27.984090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.963804    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.984090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.967223    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:27.984090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.967289    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.984090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.974136    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.984090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.974238    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:23:27.996463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:27.996413    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:23:28.046847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.014393    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.051150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.021176    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:28.051459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.051403    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.137315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.137266    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.311256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.311194    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.331459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.331404    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.340302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.335557    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.568191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.562260    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:23:28.571109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.571053    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:28.571406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.571351    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:28.711559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.703205    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.939153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.927908    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.980977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:28.980930    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.047557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.047509    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.110308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.110267    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.139191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.139142    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.158428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.158387    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:29.158703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.158671    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.367083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.366309    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.367083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.366580    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:23:29.403259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.403219    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:29.410193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.410147    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.638158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.543121    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:29.638158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.543183    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:29.638158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.570777    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.638158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.571483    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:29.638158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.571705    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.730140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.722333    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.931140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.930183    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.003122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:29.988160    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.051376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.051309    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.193207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.193100    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:23:30.345006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.201189    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.399629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.399569    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.400957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.400917    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:23:30.533526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.533498    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:23:30.533695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.533563    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:23:30.533695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.533594    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:23:30.533778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.533727    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:23:30.533778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.533756    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:23:30.533835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.533784    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:23:30.533835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.533824    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:23:30.533898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.533864    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:23:30.534482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.534467    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:23:30.539322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.539293    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:30.539535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:30.539505    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:31.539633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:31.539588    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:23:31.540038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:31.539659    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 19:23:32.860955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:32.860918    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:32.861544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:32.861437    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:32.862344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:32.862321    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:23:33.017272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.017235    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"]
Oct 30 19:23:33.022100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.022043    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.040731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.038491    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:33.040731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.038598    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:33.080874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.080816    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"]
Oct 30 19:23:33.150525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.150431    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npjzg\" (UniqueName: \"kubernetes.io/projected/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-kube-api-access-npjzg\") pod \"maas-api-revoke-sub-5a10c6ec-nnjbk\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.150702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.150542    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a10c6ec-nnjbk\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.150766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.150726    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a10c6ec-nnjbk\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.251809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.251776    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a10c6ec-nnjbk\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.251988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.251977    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a10c6ec-nnjbk\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.252460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.252210    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npjzg\" (UniqueName: \"kubernetes.io/projected/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-kube-api-access-npjzg\") pod \"maas-api-revoke-sub-5a10c6ec-nnjbk\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.252642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.252619    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a10c6ec-nnjbk\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.254429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.254408    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a10c6ec-nnjbk\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.264581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.264559    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjzg\" (UniqueName: \"kubernetes.io/projected/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-kube-api-access-npjzg\") pod \"maas-api-revoke-sub-5a10c6ec-nnjbk\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.375281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.375250    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:33.705759 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:23:33.705533    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3973d22_e097_45cc_a6d3_6fff2f5cb1da.slice/crio-21ab321f1d8794ec3bbe0a7b644882c8300d810b5b2fb53ec014dee10fefa0e6 WatchSource:0}: Error finding container 21ab321f1d8794ec3bbe0a7b644882c8300d810b5b2fb53ec014dee10fefa0e6: Status 404 returned error can't find the container with id 21ab321f1d8794ec3bbe0a7b644882c8300d810b5b2fb53ec014dee10fefa0e6
Oct 30 19:23:33.707247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:33.707218    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"]
Oct 30 19:23:34.573983 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:34.573938    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk" event={"ID":"e3973d22-e097-45cc-a6d3-6fff2f5cb1da","Type":"ContainerStarted","Data":"676b05b1790902d8b72140276c9cfda229c1532f82451e08fccbce0951976062"}
Oct 30 19:23:34.574400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:34.573999    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk" event={"ID":"e3973d22-e097-45cc-a6d3-6fff2f5cb1da","Type":"ContainerStarted","Data":"21ab321f1d8794ec3bbe0a7b644882c8300d810b5b2fb53ec014dee10fefa0e6"}
Oct 30 19:23:34.598522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:34.598471    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk" podStartSLOduration=2.598457175 podStartE2EDuration="2.598457175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:34.59504287 +0000 UTC m=+996.591958252" watchObservedRunningTime="2026-10-30 19:23:34.598457175 +0000 UTC m=+996.595372559"
Oct 30 19:23:35.046622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.046587    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"]
Oct 30 19:23:35.051581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.051556    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.072204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.066641    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"]
Oct 30 19:23:35.096270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.096218    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s97qg\" (UniqueName: \"kubernetes.io/projected/48c3aee7-aa29-46d0-848b-473e5d4d46e7-kube-api-access-s97qg\") pod \"maas-api-revoke-sub-c3c1a3ae-4x46x\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.096467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.096430    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3c1a3ae-4x46x\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.096602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.096579    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3c1a3ae-4x46x\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.198775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.198734    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3c1a3ae-4x46x\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.200759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.200731    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3c1a3ae-4x46x\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.200984 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.200969    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s97qg\" (UniqueName: \"kubernetes.io/projected/48c3aee7-aa29-46d0-848b-473e5d4d46e7-kube-api-access-s97qg\") pod \"maas-api-revoke-sub-c3c1a3ae-4x46x\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.202123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.202093    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3c1a3ae-4x46x\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.203252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.203228    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3c1a3ae-4x46x\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.214039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.214008    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s97qg\" (UniqueName: \"kubernetes.io/projected/48c3aee7-aa29-46d0-848b-473e5d4d46e7-kube-api-access-s97qg\") pod \"maas-api-revoke-sub-c3c1a3ae-4x46x\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.377089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.376987    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:35.590877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.590835    6777 generic.go:356] "Generic (PLEG): container finished" podID="e3973d22-e097-45cc-a6d3-6fff2f5cb1da" containerID="676b05b1790902d8b72140276c9cfda229c1532f82451e08fccbce0951976062" exitCode=0
Oct 30 19:23:35.591397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.590915    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk" event={"ID":"e3973d22-e097-45cc-a6d3-6fff2f5cb1da","Type":"ContainerDied","Data":"676b05b1790902d8b72140276c9cfda229c1532f82451e08fccbce0951976062"}
Oct 30 19:23:35.660135 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:23:35.660099    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c3aee7_aa29_46d0_848b_473e5d4d46e7.slice/crio-0d0252216b80449d1fcb07add3d819f9c9c21de47cf12bfd74734d787f9f5d6f WatchSource:0}: Error finding container 0d0252216b80449d1fcb07add3d819f9c9c21de47cf12bfd74734d787f9f5d6f: Status 404 returned error can't find the container with id 0d0252216b80449d1fcb07add3d819f9c9c21de47cf12bfd74734d787f9f5d6f
Oct 30 19:23:35.660696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:35.660645    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"]
Oct 30 19:23:36.601552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.601491    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x" event={"ID":"48c3aee7-aa29-46d0-848b-473e5d4d46e7","Type":"ContainerStarted","Data":"194a07df123bc668d6cb60f478da78934e1c2cbdf82e5b67f7b4ebe7e3be9729"}
Oct 30 19:23:36.601940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.601564    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x" event={"ID":"48c3aee7-aa29-46d0-848b-473e5d4d46e7","Type":"ContainerStarted","Data":"0d0252216b80449d1fcb07add3d819f9c9c21de47cf12bfd74734d787f9f5d6f"}
Oct 30 19:23:36.623991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.623930    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x" podStartSLOduration=1.6239112900000001 podStartE2EDuration="1.62391129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:36.619943174 +0000 UTC m=+998.616858562" watchObservedRunningTime="2026-10-30 19:23:36.62391129 +0000 UTC m=+998.620826678"
Oct 30 19:23:36.792532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.792496    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:36.828320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.828285    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-kube-api-access-npjzg\" (UniqueName: \"kubernetes.io/projected/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-kube-api-access-npjzg\") pod \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") "
Oct 30 19:23:36.828522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.828340    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-openshift-ca\") pod \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") "
Oct 30 19:23:36.828522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.828390    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-xks-ca\") pod \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\" (UID: \"e3973d22-e097-45cc-a6d3-6fff2f5cb1da\") "
Oct 30 19:23:36.829684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.829643    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-openshift-ca" pod "e3973d22-e097-45cc-a6d3-6fff2f5cb1da" (UID: "e3973d22-e097-45cc-a6d3-6fff2f5cb1da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:36.830875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.830854    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-kube-api-access-npjzg" pod "e3973d22-e097-45cc-a6d3-6fff2f5cb1da" (UID: "e3973d22-e097-45cc-a6d3-6fff2f5cb1da"). InnerVolumeSpecName "kube-api-access-npjzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:36.836472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.836445    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-xks-ca" pod "e3973d22-e097-45cc-a6d3-6fff2f5cb1da" (UID: "e3973d22-e097-45cc-a6d3-6fff2f5cb1da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:36.930997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.930905    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npjzg\" (UniqueName: \"kubernetes.io/projected/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-kube-api-access-npjzg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:36.930997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.930934    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:36.930997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:36.930944    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3973d22-e097-45cc-a6d3-6fff2f5cb1da-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:37.611420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:37.611389    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"
Oct 30 19:23:37.611420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:37.611414    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk" event={"ID":"e3973d22-e097-45cc-a6d3-6fff2f5cb1da","Type":"ContainerDied","Data":"21ab321f1d8794ec3bbe0a7b644882c8300d810b5b2fb53ec014dee10fefa0e6"}
Oct 30 19:23:37.611921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:37.611469    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21ab321f1d8794ec3bbe0a7b644882c8300d810b5b2fb53ec014dee10fefa0e6"
Oct 30 19:23:37.613202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:37.613176    6777 generic.go:356] "Generic (PLEG): container finished" podID="48c3aee7-aa29-46d0-848b-473e5d4d46e7" containerID="194a07df123bc668d6cb60f478da78934e1c2cbdf82e5b67f7b4ebe7e3be9729" exitCode=0
Oct 30 19:23:37.613316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:37.613218    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x" event={"ID":"48c3aee7-aa29-46d0-848b-473e5d4d46e7","Type":"ContainerDied","Data":"194a07df123bc668d6cb60f478da78934e1c2cbdf82e5b67f7b4ebe7e3be9729"}
Oct 30 19:23:38.816750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.816723    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:38.865015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.864978    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c3aee7-aa29-46d0-848b-473e5d4d46e7-kube-api-access-s97qg\" (UniqueName: \"kubernetes.io/projected/48c3aee7-aa29-46d0-848b-473e5d4d46e7-kube-api-access-s97qg\") pod \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") "
Oct 30 19:23:38.865208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.865121    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-openshift-ca\") pod \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") "
Oct 30 19:23:38.865208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.865169    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-xks-ca\") pod \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\" (UID: \"48c3aee7-aa29-46d0-848b-473e5d4d46e7\") "
Oct 30 19:23:38.865568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.865532    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-openshift-ca" pod "48c3aee7-aa29-46d0-848b-473e5d4d46e7" (UID: "48c3aee7-aa29-46d0-848b-473e5d4d46e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:38.865987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.865966    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:38.867375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.867349    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-xks-ca" pod "48c3aee7-aa29-46d0-848b-473e5d4d46e7" (UID: "48c3aee7-aa29-46d0-848b-473e5d4d46e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:38.867735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.867713    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c3aee7-aa29-46d0-848b-473e5d4d46e7-kube-api-access-s97qg" pod "48c3aee7-aa29-46d0-848b-473e5d4d46e7" (UID: "48c3aee7-aa29-46d0-848b-473e5d4d46e7"). InnerVolumeSpecName "kube-api-access-s97qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:38.967843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.967802    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s97qg\" (UniqueName: \"kubernetes.io/projected/48c3aee7-aa29-46d0-848b-473e5d4d46e7-kube-api-access-s97qg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:38.967843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:38.967835    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c3aee7-aa29-46d0-848b-473e5d4d46e7-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:39.630712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:39.630684    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"
Oct 30 19:23:39.630898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:39.630684    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x" event={"ID":"48c3aee7-aa29-46d0-848b-473e5d4d46e7","Type":"ContainerDied","Data":"0d0252216b80449d1fcb07add3d819f9c9c21de47cf12bfd74734d787f9f5d6f"}
Oct 30 19:23:39.630898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:39.630816    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d0252216b80449d1fcb07add3d819f9c9c21de47cf12bfd74734d787f9f5d6f"
Oct 30 19:23:43.640914 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:23:43.640873    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:23:46.344085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:46.343169    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.371082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:46.370164    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.512368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:46.511156    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:23:46.563594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:46.563551    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:23:46.922245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:46.922192    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:46.922946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:46.922603    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.931607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:46.931561    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.064084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.058225    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.065212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.065173    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.334408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.334351    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.343494 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.343445    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.363116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.349366    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:23:47.366124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.356026    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.375187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.375143    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.384407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.375810    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:23:47.584079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.580654    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.584079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.580981    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:23:47.584079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.581329    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:23:47.584079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.581610    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:23:47.584079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.581667    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:23:47.955083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.954786    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Oct 30 19:23:47.955316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.955160    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:23:47.955316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.955274    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Oct 30 19:23:47.998091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:47.997722    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded"
Oct 30 19:23:48.014098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.013127    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:23:48.071103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.068287    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.071103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.068940    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:23:48.136087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.134216    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.402116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.350705    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:23:48.402116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.356522    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.402116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.357353    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Oct 30 19:23:48.402116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.399159    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:23:48.402116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.399485    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:23:48.583471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.583418    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.584844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.584804    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.702147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.698753    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:23:48.702147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.698831    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:23:48.702147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:48.698870    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:23:49.048359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.048315    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:23:49.074084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.070622    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.115106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.076270    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:49.115389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.115356    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.534082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.533394    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.632490 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.632411    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.696087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.645458    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:23:49.696087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.645913    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:23:49.746296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.746250    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.779410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.779356    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.780420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:49.780382    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:23:50.055108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:50.052204    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.067636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:50.067598    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:23:50.147968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:50.147914    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.553089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:50.552167    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:23:50.663083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:50.656719    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.663083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:50.658654    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:23:50.776383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:50.776328    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:23:50.918145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:50.836362    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.918439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:50.868724    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:51.075094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:51.074196    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:23:51.221479 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:51.221390    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:23:51.264131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:51.258587    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:23:51.264466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:51.264444    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:23:51.264603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:51.264594    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:23:51.317642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:51.307585    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:23:51.317642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:51.316650    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:23:51.324280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:51.324253    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:23:51.324905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:51.324879    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:23:53.320289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.320252    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"]
Oct 30 19:23:53.321233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.321205    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e3973d22-e097-45cc-a6d3-6fff2f5cb1da" containerName="revoke-keys"
Oct 30 19:23:53.321233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.321231    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3973d22-e097-45cc-a6d3-6fff2f5cb1da" containerName="revoke-keys"
Oct 30 19:23:53.321448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.321258    6777 cpu_manager.go:414] "containerMap: removing container" podUID="48c3aee7-aa29-46d0-848b-473e5d4d46e7" containerName="revoke-keys"
Oct 30 19:23:53.321448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.321268    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c3aee7-aa29-46d0-848b-473e5d4d46e7" containerName="revoke-keys"
Oct 30 19:23:53.321552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.321487    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3973d22-e097-45cc-a6d3-6fff2f5cb1da" containerName="revoke-keys"
Oct 30 19:23:53.321552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.321506    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c3aee7-aa29-46d0-848b-473e5d4d46e7" containerName="revoke-keys"
Oct 30 19:23:53.325945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.325920    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.331545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.331510    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:53.331718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.331559    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:53.334326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.334292    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"]
Oct 30 19:23:53.432169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.432126    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf931055-5000-4c9e-93ff-06c113435975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-697c73c2-q6hsr\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.432375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.432342    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf931055-5000-4c9e-93ff-06c113435975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-697c73c2-q6hsr\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.432447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.432433    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2k6\" (UniqueName: \"kubernetes.io/projected/cf931055-5000-4c9e-93ff-06c113435975-kube-api-access-xs2k6\") pod \"maas-api-revoke-sub-697c73c2-q6hsr\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.533428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.533391    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf931055-5000-4c9e-93ff-06c113435975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-697c73c2-q6hsr\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.533614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.533482    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2k6\" (UniqueName: \"kubernetes.io/projected/cf931055-5000-4c9e-93ff-06c113435975-kube-api-access-xs2k6\") pod \"maas-api-revoke-sub-697c73c2-q6hsr\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.533657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.533640    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf931055-5000-4c9e-93ff-06c113435975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-697c73c2-q6hsr\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.534283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.534255    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf931055-5000-4c9e-93ff-06c113435975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-697c73c2-q6hsr\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.536268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.536244    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf931055-5000-4c9e-93ff-06c113435975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-697c73c2-q6hsr\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.546609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.546581    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2k6\" (UniqueName: \"kubernetes.io/projected/cf931055-5000-4c9e-93ff-06c113435975-kube-api-access-xs2k6\") pod \"maas-api-revoke-sub-697c73c2-q6hsr\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.655270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.655177    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:53.943325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:53.943287    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"]
Oct 30 19:23:53.944404 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:23:53.944378    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf931055_5000_4c9e_93ff_06c113435975.slice/crio-be41aec4e9a6f830ff40cf6996f4200d6795a41fe206525a30e90ad23651726f WatchSource:0}: Error finding container be41aec4e9a6f830ff40cf6996f4200d6795a41fe206525a30e90ad23651726f: Status 404 returned error can't find the container with id be41aec4e9a6f830ff40cf6996f4200d6795a41fe206525a30e90ad23651726f
Oct 30 19:23:54.338452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:54.338415    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr" event={"ID":"cf931055-5000-4c9e-93ff-06c113435975","Type":"ContainerStarted","Data":"be41aec4e9a6f830ff40cf6996f4200d6795a41fe206525a30e90ad23651726f"}
Oct 30 19:23:55.348629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:55.348593    6777 generic.go:356] "Generic (PLEG): container finished" podID="cf931055-5000-4c9e-93ff-06c113435975" containerID="c2de734de49df6925f63e14963329d8091b7e3abb2d361f2c927a5c2cd815cc7" exitCode=0
Oct 30 19:23:55.349053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:55.348646    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr" event={"ID":"cf931055-5000-4c9e-93ff-06c113435975","Type":"ContainerDied","Data":"c2de734de49df6925f63e14963329d8091b7e3abb2d361f2c927a5c2cd815cc7"}
Oct 30 19:23:56.549352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.549320    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:56.700113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.700015    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf931055-5000-4c9e-93ff-06c113435975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf931055-5000-4c9e-93ff-06c113435975-maas-api-openshift-ca\") pod \"cf931055-5000-4c9e-93ff-06c113435975\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") "
Oct 30 19:23:56.700299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.700158    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf931055-5000-4c9e-93ff-06c113435975-kube-api-access-xs2k6\" (UniqueName: \"kubernetes.io/projected/cf931055-5000-4c9e-93ff-06c113435975-kube-api-access-xs2k6\") pod \"cf931055-5000-4c9e-93ff-06c113435975\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") "
Oct 30 19:23:56.700299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.700211    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf931055-5000-4c9e-93ff-06c113435975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf931055-5000-4c9e-93ff-06c113435975-maas-api-xks-ca\") pod \"cf931055-5000-4c9e-93ff-06c113435975\" (UID: \"cf931055-5000-4c9e-93ff-06c113435975\") "
Oct 30 19:23:56.700441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.700415    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf931055-5000-4c9e-93ff-06c113435975-maas-api-openshift-ca" pod "cf931055-5000-4c9e-93ff-06c113435975" (UID: "cf931055-5000-4c9e-93ff-06c113435975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:56.700831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.700816    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf931055-5000-4c9e-93ff-06c113435975-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:56.702257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.702235    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf931055-5000-4c9e-93ff-06c113435975-maas-api-xks-ca" pod "cf931055-5000-4c9e-93ff-06c113435975" (UID: "cf931055-5000-4c9e-93ff-06c113435975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:56.702325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.702303    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf931055-5000-4c9e-93ff-06c113435975-kube-api-access-xs2k6" pod "cf931055-5000-4c9e-93ff-06c113435975" (UID: "cf931055-5000-4c9e-93ff-06c113435975"). InnerVolumeSpecName "kube-api-access-xs2k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:56.802724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.802685    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2k6\" (UniqueName: \"kubernetes.io/projected/cf931055-5000-4c9e-93ff-06c113435975-kube-api-access-xs2k6\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:56.802724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:56.802720    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf931055-5000-4c9e-93ff-06c113435975-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:57.366011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:57.365975    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"
Oct 30 19:23:57.366011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:57.365993    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr" event={"ID":"cf931055-5000-4c9e-93ff-06c113435975","Type":"ContainerDied","Data":"be41aec4e9a6f830ff40cf6996f4200d6795a41fe206525a30e90ad23651726f"}
Oct 30 19:23:57.366259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:57.366036    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be41aec4e9a6f830ff40cf6996f4200d6795a41fe206525a30e90ad23651726f"
Oct 30 19:23:58.744352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.744317    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"]
Oct 30 19:23:58.745285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.745265    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cf931055-5000-4c9e-93ff-06c113435975" containerName="revoke-keys"
Oct 30 19:23:58.745363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.745286    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf931055-5000-4c9e-93ff-06c113435975" containerName="revoke-keys"
Oct 30 19:23:58.745513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.745500    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf931055-5000-4c9e-93ff-06c113435975" containerName="revoke-keys"
Oct 30 19:23:58.750039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.750014    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:58.755231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.755199    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"]
Oct 30 19:23:58.755967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.755941    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:58.756139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.755997    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:58.832570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.832534    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstv7\" (UniqueName: \"kubernetes.io/projected/549e3e83-1f92-4842-854e-0cf02b70294a-kube-api-access-rstv7\") pod \"maas-api-revoke-sub-1a160f2d-49w89\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:58.832772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.832665    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a160f2d-49w89\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:58.832934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.832911    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a160f2d-49w89\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:58.934292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.934253    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a160f2d-49w89\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:58.934493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.934393    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstv7\" (UniqueName: \"kubernetes.io/projected/549e3e83-1f92-4842-854e-0cf02b70294a-kube-api-access-rstv7\") pod \"maas-api-revoke-sub-1a160f2d-49w89\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:58.934493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.934440    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a160f2d-49w89\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:58.934894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.934873    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a160f2d-49w89\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:58.937074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.937036    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a160f2d-49w89\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:58.947222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:58.947197    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstv7\" (UniqueName: \"kubernetes.io/projected/549e3e83-1f92-4842-854e-0cf02b70294a-kube-api-access-rstv7\") pod \"maas-api-revoke-sub-1a160f2d-49w89\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:59.071536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:59.071499    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:23:59.373883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:23:59.373850    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"]
Oct 30 19:23:59.378171 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:23:59.378141    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549e3e83_1f92_4842_854e_0cf02b70294a.slice/crio-8cab6a0a2cf7835e59e270f47075fe91ebbc121db9a12de9c779c60652b92171 WatchSource:0}: Error finding container 8cab6a0a2cf7835e59e270f47075fe91ebbc121db9a12de9c779c60652b92171: Status 404 returned error can't find the container with id 8cab6a0a2cf7835e59e270f47075fe91ebbc121db9a12de9c779c60652b92171
Oct 30 19:24:00.393800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:00.393764    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89" event={"ID":"549e3e83-1f92-4842-854e-0cf02b70294a","Type":"ContainerStarted","Data":"f932186080a30b4b86f62ea8e3d7e265dc2307022ebcf8cf6136bee53df42df1"}
Oct 30 19:24:00.394199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:00.393823    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89" event={"ID":"549e3e83-1f92-4842-854e-0cf02b70294a","Type":"ContainerStarted","Data":"8cab6a0a2cf7835e59e270f47075fe91ebbc121db9a12de9c779c60652b92171"}
Oct 30 19:24:00.418372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:00.418325    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89" podStartSLOduration=2.418312089 podStartE2EDuration="2.418312089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:00.417557813 +0000 UTC m=+1022.414473196" watchObservedRunningTime="2026-10-30 19:24:00.418312089 +0000 UTC m=+1022.415227473"
Oct 30 19:24:00.769950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:00.769862    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:24:01.406047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:01.406009    6777 generic.go:356] "Generic (PLEG): container finished" podID="549e3e83-1f92-4842-854e-0cf02b70294a" containerID="f932186080a30b4b86f62ea8e3d7e265dc2307022ebcf8cf6136bee53df42df1" exitCode=0
Oct 30 19:24:01.406504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:01.406092    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89" event={"ID":"549e3e83-1f92-4842-854e-0cf02b70294a","Type":"ContainerDied","Data":"f932186080a30b4b86f62ea8e3d7e265dc2307022ebcf8cf6136bee53df42df1"}
Oct 30 19:24:03.472306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:03.472157    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.560333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:03.550374    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.668647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:03.668593    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.759100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:03.670119    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.759100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:03.670185    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:24:03.759100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:03.670263    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.759100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:03.670286    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Oct 30 19:24:03.935306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:03.935190    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:04.198998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.198784    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.349186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.337516    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.487201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.482276    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.487201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.482384    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:24:04.604694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.579174    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:24:04.740467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.740360    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.759312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.759223    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.798280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.740296    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.798753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.798538    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.012204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:04.975402    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.057098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.042680    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:24:05.057098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.042753    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:24:05.057098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.042792    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:24:05.057098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.042828    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:24:05.057098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.042862    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:24:05.057098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.042896    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:24:05.057098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.042930    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:24:05.099564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.099533    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:24:05.327155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.112912    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-xks-ca\") pod \"549e3e83-1f92-4842-854e-0cf02b70294a\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") "
Oct 30 19:24:05.327155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.238614    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-openshift-ca\") pod \"549e3e83-1f92-4842-854e-0cf02b70294a\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") "
Oct 30 19:24:05.327155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.238701    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549e3e83-1f92-4842-854e-0cf02b70294a-kube-api-access-rstv7\" (UniqueName: \"kubernetes.io/projected/549e3e83-1f92-4842-854e-0cf02b70294a-kube-api-access-rstv7\") pod \"549e3e83-1f92-4842-854e-0cf02b70294a\" (UID: \"549e3e83-1f92-4842-854e-0cf02b70294a\") "
Oct 30 19:24:05.327155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.248207    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-openshift-ca" pod "549e3e83-1f92-4842-854e-0cf02b70294a" (UID: "549e3e83-1f92-4842-854e-0cf02b70294a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:05.327155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.248504    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:05.327155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.254309    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:24:05.327155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.315822    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:05.396107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.341198    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.396107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.380963    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:24:05.471757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.454231    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549e3e83-1f92-4842-854e-0cf02b70294a-kube-api-access-rstv7" pod "549e3e83-1f92-4842-854e-0cf02b70294a" (UID: "549e3e83-1f92-4842-854e-0cf02b70294a"). InnerVolumeSpecName "kube-api-access-rstv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:05.477724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.476240    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-xks-ca" pod "549e3e83-1f92-4842-854e-0cf02b70294a" (UID: "549e3e83-1f92-4842-854e-0cf02b70294a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:05.483830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.483789    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.540729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.540688    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549e3e83-1f92-4842-854e-0cf02b70294a-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:05.543591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.543560    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstv7\" (UniqueName: \"kubernetes.io/projected/549e3e83-1f92-4842-854e-0cf02b70294a-kube-api-access-rstv7\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:05.703145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.625175    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:24:05.764200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.764138    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:05.768878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.768471    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:24:05.873122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.826205    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:24:05.873122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.843174    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.015138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:05.976156    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:06.210084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.193904    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"
Oct 30 19:24:06.257081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.256996    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.326088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.322374    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89" event={"ID":"549e3e83-1f92-4842-854e-0cf02b70294a","Type":"ContainerDied","Data":"8cab6a0a2cf7835e59e270f47075fe91ebbc121db9a12de9c779c60652b92171"}
Oct 30 19:24:06.326391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.326365    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cab6a0a2cf7835e59e270f47075fe91ebbc121db9a12de9c779c60652b92171"
Oct 30 19:24:06.343088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.342086    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.496225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.496180    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:24:06.499588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.499537    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:24:06.522352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.522324    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:24:06.642867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.637320    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:24:06.710398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.709398    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:24:06.785076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.776833    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:24:06.807637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.807494    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:24:06.809696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.809667    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.829633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.829593    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:24:06.872786 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:24:06.872746    6777 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549e3e83_1f92_4842_854e_0cf02b70294a.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:24:06.887592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.887562    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:24:06.890976 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.890927    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:24:06.901173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.894931    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:24:06.928250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.928190    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"]
Oct 30 19:24:06.929756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.929732    6777 cpu_manager.go:414] "containerMap: removing container" podUID="549e3e83-1f92-4842-854e-0cf02b70294a" containerName="revoke-keys"
Oct 30 19:24:06.929756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.929755    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549e3e83-1f92-4842-854e-0cf02b70294a" containerName="revoke-keys"
Oct 30 19:24:06.930102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.930087    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="549e3e83-1f92-4842-854e-0cf02b70294a" containerName="revoke-keys"
Oct 30 19:24:06.948318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.948286    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"]
Oct 30 19:24:06.948551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.948435    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:06.955258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.955229    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:06.955258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:06.955244    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:24:07.004117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.004074    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a71dd47f-v6x86\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.004475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.004451    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvfh\" (UniqueName: \"kubernetes.io/projected/4a0ae996-8882-4ef1-8dc1-0d242347943d-kube-api-access-6lvfh\") pod \"maas-api-revoke-sub-a71dd47f-v6x86\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.004587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.004489    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a71dd47f-v6x86\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.106340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.106307    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvfh\" (UniqueName: \"kubernetes.io/projected/4a0ae996-8882-4ef1-8dc1-0d242347943d-kube-api-access-6lvfh\") pod \"maas-api-revoke-sub-a71dd47f-v6x86\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.106510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.106344    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a71dd47f-v6x86\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.106510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.106424    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a71dd47f-v6x86\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.107073 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.107027    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a71dd47f-v6x86\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.109025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.109004    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a71dd47f-v6x86\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.119114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.119094    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvfh\" (UniqueName: \"kubernetes.io/projected/4a0ae996-8882-4ef1-8dc1-0d242347943d-kube-api-access-6lvfh\") pod \"maas-api-revoke-sub-a71dd47f-v6x86\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.267417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.267336    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:07.558243 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:07.558208    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0ae996_8882_4ef1_8dc1_0d242347943d.slice/crio-66a714946c328fbdbf1deca9b9ac440da37f5ef5e695d5b8b41a460f042e8a2a WatchSource:0}: Error finding container 66a714946c328fbdbf1deca9b9ac440da37f5ef5e695d5b8b41a460f042e8a2a: Status 404 returned error can't find the container with id 66a714946c328fbdbf1deca9b9ac440da37f5ef5e695d5b8b41a460f042e8a2a
Oct 30 19:24:07.563398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:07.563371    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"]
Oct 30 19:24:08.280682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:08.280639    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86" event={"ID":"4a0ae996-8882-4ef1-8dc1-0d242347943d","Type":"ContainerStarted","Data":"0f9c6777623b571847ae79383998ce8ceec1f40e61edf6c68cf2365293973025"}
Oct 30 19:24:08.281174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:08.280702    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86" event={"ID":"4a0ae996-8882-4ef1-8dc1-0d242347943d","Type":"ContainerStarted","Data":"66a714946c328fbdbf1deca9b9ac440da37f5ef5e695d5b8b41a460f042e8a2a"}
Oct 30 19:24:08.303048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:08.302975    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86" podStartSLOduration=2.302954078 podStartE2EDuration="2.302954078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:08.299395887 +0000 UTC m=+1030.296311273" watchObservedRunningTime="2026-10-30 19:24:08.302954078 +0000 UTC m=+1030.299869486"
Oct 30 19:24:09.290501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:09.290468    6777 generic.go:356] "Generic (PLEG): container finished" podID="4a0ae996-8882-4ef1-8dc1-0d242347943d" containerID="0f9c6777623b571847ae79383998ce8ceec1f40e61edf6c68cf2365293973025" exitCode=0
Oct 30 19:24:09.290923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:09.290508    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86" event={"ID":"4a0ae996-8882-4ef1-8dc1-0d242347943d","Type":"ContainerDied","Data":"0f9c6777623b571847ae79383998ce8ceec1f40e61edf6c68cf2365293973025"}
Oct 30 19:24:10.453982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.453955    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:10.570115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.569979    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-openshift-ca\") pod \"4a0ae996-8882-4ef1-8dc1-0d242347943d\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") "
Oct 30 19:24:10.570311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.570290    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0ae996-8882-4ef1-8dc1-0d242347943d-kube-api-access-6lvfh\" (UniqueName: \"kubernetes.io/projected/4a0ae996-8882-4ef1-8dc1-0d242347943d-kube-api-access-6lvfh\") pod \"4a0ae996-8882-4ef1-8dc1-0d242347943d\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") "
Oct 30 19:24:10.570382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.570355    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-openshift-ca" pod "4a0ae996-8882-4ef1-8dc1-0d242347943d" (UID: "4a0ae996-8882-4ef1-8dc1-0d242347943d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:10.570447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.570359    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-xks-ca\") pod \"4a0ae996-8882-4ef1-8dc1-0d242347943d\" (UID: \"4a0ae996-8882-4ef1-8dc1-0d242347943d\") "
Oct 30 19:24:10.570942 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.570924    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:10.572409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.572381    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-xks-ca" pod "4a0ae996-8882-4ef1-8dc1-0d242347943d" (UID: "4a0ae996-8882-4ef1-8dc1-0d242347943d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:10.572503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.572445    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0ae996-8882-4ef1-8dc1-0d242347943d-kube-api-access-6lvfh" pod "4a0ae996-8882-4ef1-8dc1-0d242347943d" (UID: "4a0ae996-8882-4ef1-8dc1-0d242347943d"). InnerVolumeSpecName "kube-api-access-6lvfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:10.672395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.672360    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lvfh\" (UniqueName: \"kubernetes.io/projected/4a0ae996-8882-4ef1-8dc1-0d242347943d-kube-api-access-6lvfh\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:10.672395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:10.672390    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0ae996-8882-4ef1-8dc1-0d242347943d-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:11.306491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:11.306453    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"
Oct 30 19:24:11.306491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:11.306480    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86" event={"ID":"4a0ae996-8882-4ef1-8dc1-0d242347943d","Type":"ContainerDied","Data":"66a714946c328fbdbf1deca9b9ac440da37f5ef5e695d5b8b41a460f042e8a2a"}
Oct 30 19:24:11.306689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:11.306525    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a714946c328fbdbf1deca9b9ac440da37f5ef5e695d5b8b41a460f042e8a2a"
Oct 30 19:24:13.650454 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:24:13.650409    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:24:16.951286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:16.951232    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"]
Oct 30 19:24:16.951923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:16.951905    6777 cpu_manager.go:414] "containerMap: removing container" podUID="4a0ae996-8882-4ef1-8dc1-0d242347943d" containerName="revoke-keys"
Oct 30 19:24:16.951971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:16.951924    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0ae996-8882-4ef1-8dc1-0d242347943d" containerName="revoke-keys"
Oct 30 19:24:16.952079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:16.952053    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0ae996-8882-4ef1-8dc1-0d242347943d" containerName="revoke-keys"
Oct 30 19:24:16.955844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:16.955817    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:16.961010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:16.960978    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:16.961203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:16.960984    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:24:16.966000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:16.965966    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"]
Oct 30 19:24:17.086774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.086730    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85a43681-tq2xf\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.087043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.086887    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85a43681-tq2xf\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.087043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.086975    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m4f2\" (UniqueName: \"kubernetes.io/projected/20de73d9-3deb-45dd-8294-ecbfc76bbd27-kube-api-access-8m4f2\") pod \"maas-api-revoke-sub-85a43681-tq2xf\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.188472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.188429    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85a43681-tq2xf\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.188823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.188531    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m4f2\" (UniqueName: \"kubernetes.io/projected/20de73d9-3deb-45dd-8294-ecbfc76bbd27-kube-api-access-8m4f2\") pod \"maas-api-revoke-sub-85a43681-tq2xf\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.188823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.188663    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85a43681-tq2xf\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.189325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.189305    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85a43681-tq2xf\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.191029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.190986    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85a43681-tq2xf\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.201609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.201541    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m4f2\" (UniqueName: \"kubernetes.io/projected/20de73d9-3deb-45dd-8294-ecbfc76bbd27-kube-api-access-8m4f2\") pod \"maas-api-revoke-sub-85a43681-tq2xf\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.275638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.275593    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:17.571969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:17.571938    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"]
Oct 30 19:24:17.572146 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:17.572046    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20de73d9_3deb_45dd_8294_ecbfc76bbd27.slice/crio-f29e2efd16d19b0d58a35e8b02bc20172f413a4871079c4d8d2c79265c6c0c71 WatchSource:0}: Error finding container f29e2efd16d19b0d58a35e8b02bc20172f413a4871079c4d8d2c79265c6c0c71: Status 404 returned error can't find the container with id f29e2efd16d19b0d58a35e8b02bc20172f413a4871079c4d8d2c79265c6c0c71
Oct 30 19:24:18.372832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:18.372794    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf" event={"ID":"20de73d9-3deb-45dd-8294-ecbfc76bbd27","Type":"ContainerStarted","Data":"02205bfe7e9c96c0eeb3eb75ad61ac776e3ca2f5a68b94a71b4680454c003a9b"}
Oct 30 19:24:18.373287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:18.372857    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf" event={"ID":"20de73d9-3deb-45dd-8294-ecbfc76bbd27","Type":"ContainerStarted","Data":"f29e2efd16d19b0d58a35e8b02bc20172f413a4871079c4d8d2c79265c6c0c71"}
Oct 30 19:24:18.398211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:18.398161    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf" podStartSLOduration=2.398148015 podStartE2EDuration="2.398148015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:18.395797348 +0000 UTC m=+1040.392712733" watchObservedRunningTime="2026-10-30 19:24:18.398148015 +0000 UTC m=+1040.395063398"
Oct 30 19:24:19.382947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:19.382911    6777 generic.go:356] "Generic (PLEG): container finished" podID="20de73d9-3deb-45dd-8294-ecbfc76bbd27" containerID="02205bfe7e9c96c0eeb3eb75ad61ac776e3ca2f5a68b94a71b4680454c003a9b" exitCode=0
Oct 30 19:24:19.383516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:19.383023    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf" event={"ID":"20de73d9-3deb-45dd-8294-ecbfc76bbd27","Type":"ContainerDied","Data":"02205bfe7e9c96c0eeb3eb75ad61ac776e3ca2f5a68b94a71b4680454c003a9b"}
Oct 30 19:24:20.557095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.557042    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:20.643845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.643770    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20de73d9-3deb-45dd-8294-ecbfc76bbd27-kube-api-access-8m4f2\" (UniqueName: \"kubernetes.io/projected/20de73d9-3deb-45dd-8294-ecbfc76bbd27-kube-api-access-8m4f2\") pod \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") "
Oct 30 19:24:20.643845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.643839    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-xks-ca\") pod \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") "
Oct 30 19:24:20.644034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.644003    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-openshift-ca\") pod \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\" (UID: \"20de73d9-3deb-45dd-8294-ecbfc76bbd27\") "
Oct 30 19:24:20.644360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.644333    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-openshift-ca" pod "20de73d9-3deb-45dd-8294-ecbfc76bbd27" (UID: "20de73d9-3deb-45dd-8294-ecbfc76bbd27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:20.644648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.644631    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.645959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.645933    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-xks-ca" pod "20de73d9-3deb-45dd-8294-ecbfc76bbd27" (UID: "20de73d9-3deb-45dd-8294-ecbfc76bbd27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:20.646090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.645977    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20de73d9-3deb-45dd-8294-ecbfc76bbd27-kube-api-access-8m4f2" pod "20de73d9-3deb-45dd-8294-ecbfc76bbd27" (UID: "20de73d9-3deb-45dd-8294-ecbfc76bbd27"). InnerVolumeSpecName "kube-api-access-8m4f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:20.745832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.745790    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m4f2\" (UniqueName: \"kubernetes.io/projected/20de73d9-3deb-45dd-8294-ecbfc76bbd27-kube-api-access-8m4f2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.745832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.745833    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20de73d9-3deb-45dd-8294-ecbfc76bbd27-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.770557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.770517    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"]
Oct 30 19:24:20.771312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.771295    6777 cpu_manager.go:414] "containerMap: removing container" podUID="20de73d9-3deb-45dd-8294-ecbfc76bbd27" containerName="revoke-keys"
Oct 30 19:24:20.771312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.771312    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20de73d9-3deb-45dd-8294-ecbfc76bbd27" containerName="revoke-keys"
Oct 30 19:24:20.771438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.771415    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="20de73d9-3deb-45dd-8294-ecbfc76bbd27" containerName="revoke-keys"
Oct 30 19:24:20.775253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.775233    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:20.784333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.784308    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"]
Oct 30 19:24:20.847495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.847457    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c759790d-n9p5w\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:20.847689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.847577    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c759790d-n9p5w\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:20.847818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.847801    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqgwp\" (UniqueName: \"kubernetes.io/projected/acbb908a-4d2a-4f12-b556-78d023e6789d-kube-api-access-hqgwp\") pod \"maas-api-revoke-sub-c759790d-n9p5w\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:20.949995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.949886    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c759790d-n9p5w\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:20.950201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.949988    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c759790d-n9p5w\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:20.950276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.950218    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqgwp\" (UniqueName: \"kubernetes.io/projected/acbb908a-4d2a-4f12-b556-78d023e6789d-kube-api-access-hqgwp\") pod \"maas-api-revoke-sub-c759790d-n9p5w\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:20.950992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.950972    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c759790d-n9p5w\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:20.952758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.952731    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c759790d-n9p5w\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:20.962840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:20.962822    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqgwp\" (UniqueName: \"kubernetes.io/projected/acbb908a-4d2a-4f12-b556-78d023e6789d-kube-api-access-hqgwp\") pod \"maas-api-revoke-sub-c759790d-n9p5w\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:21.099153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:21.099110    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:21.391045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:21.391013    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"]
Oct 30 19:24:21.391202 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:21.391012    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbb908a_4d2a_4f12_b556_78d023e6789d.slice/crio-8448157b0de897646ffbef6b2cfdc33a71e83d18f20a97b8389a606182b85b74 WatchSource:0}: Error finding container 8448157b0de897646ffbef6b2cfdc33a71e83d18f20a97b8389a606182b85b74: Status 404 returned error can't find the container with id 8448157b0de897646ffbef6b2cfdc33a71e83d18f20a97b8389a606182b85b74
Oct 30 19:24:21.399854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:21.399830    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"
Oct 30 19:24:21.399956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:21.399849    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf" event={"ID":"20de73d9-3deb-45dd-8294-ecbfc76bbd27","Type":"ContainerDied","Data":"f29e2efd16d19b0d58a35e8b02bc20172f413a4871079c4d8d2c79265c6c0c71"}
Oct 30 19:24:21.399956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:21.399896    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f29e2efd16d19b0d58a35e8b02bc20172f413a4871079c4d8d2c79265c6c0c71"
Oct 30 19:24:22.408868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:22.408831    6777 generic.go:356] "Generic (PLEG): container finished" podID="acbb908a-4d2a-4f12-b556-78d023e6789d" containerID="7dc9699f22f92746c824aabce8e11c86e030f6fa9b6f0f255dc4b01354bc9619" exitCode=0
Oct 30 19:24:22.409312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:22.408919    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w" event={"ID":"acbb908a-4d2a-4f12-b556-78d023e6789d","Type":"ContainerDied","Data":"7dc9699f22f92746c824aabce8e11c86e030f6fa9b6f0f255dc4b01354bc9619"}
Oct 30 19:24:22.409312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:22.408982    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w" event={"ID":"acbb908a-4d2a-4f12-b556-78d023e6789d","Type":"ContainerStarted","Data":"8448157b0de897646ffbef6b2cfdc33a71e83d18f20a97b8389a606182b85b74"}
Oct 30 19:24:23.610482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.610460    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:23.721628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.721542    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-openshift-ca\") pod \"acbb908a-4d2a-4f12-b556-78d023e6789d\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") "
Oct 30 19:24:23.721628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.721585    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbb908a-4d2a-4f12-b556-78d023e6789d-kube-api-access-hqgwp\" (UniqueName: \"kubernetes.io/projected/acbb908a-4d2a-4f12-b556-78d023e6789d-kube-api-access-hqgwp\") pod \"acbb908a-4d2a-4f12-b556-78d023e6789d\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") "
Oct 30 19:24:23.721818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.721791    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-xks-ca\") pod \"acbb908a-4d2a-4f12-b556-78d023e6789d\" (UID: \"acbb908a-4d2a-4f12-b556-78d023e6789d\") "
Oct 30 19:24:23.721956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.721934    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-openshift-ca" pod "acbb908a-4d2a-4f12-b556-78d023e6789d" (UID: "acbb908a-4d2a-4f12-b556-78d023e6789d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:23.722452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.722434    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:23.723703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.723683    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-xks-ca" pod "acbb908a-4d2a-4f12-b556-78d023e6789d" (UID: "acbb908a-4d2a-4f12-b556-78d023e6789d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:23.723769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.723714    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbb908a-4d2a-4f12-b556-78d023e6789d-kube-api-access-hqgwp" pod "acbb908a-4d2a-4f12-b556-78d023e6789d" (UID: "acbb908a-4d2a-4f12-b556-78d023e6789d"). InnerVolumeSpecName "kube-api-access-hqgwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:23.823609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.823575    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqgwp\" (UniqueName: \"kubernetes.io/projected/acbb908a-4d2a-4f12-b556-78d023e6789d-kube-api-access-hqgwp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:23.823609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:23.823605    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acbb908a-4d2a-4f12-b556-78d023e6789d-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:24.425940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:24.425860    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"
Oct 30 19:24:24.425940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:24.425887    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w" event={"ID":"acbb908a-4d2a-4f12-b556-78d023e6789d","Type":"ContainerDied","Data":"8448157b0de897646ffbef6b2cfdc33a71e83d18f20a97b8389a606182b85b74"}
Oct 30 19:24:24.426151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:24.425967    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8448157b0de897646ffbef6b2cfdc33a71e83d18f20a97b8389a606182b85b74"
Oct 30 19:24:26.566083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.560475    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"]
Oct 30 19:24:26.566083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.561511    6777 cpu_manager.go:414] "containerMap: removing container" podUID="acbb908a-4d2a-4f12-b556-78d023e6789d" containerName="revoke-keys"
Oct 30 19:24:26.566083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.561534    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acbb908a-4d2a-4f12-b556-78d023e6789d" containerName="revoke-keys"
Oct 30 19:24:26.566083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.561759    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="acbb908a-4d2a-4f12-b556-78d023e6789d" containerName="revoke-keys"
Oct 30 19:24:26.596468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.595380    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:26.643190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.625717    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:24:26.643190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.625798    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:26.701413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.701329    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"]
Oct 30 19:24:26.770098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.751433    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f2d70b7-9nd8k\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:26.782329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.770766    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f2d70b7-9nd8k\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:26.782740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.782701    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d98b8\" (UniqueName: \"kubernetes.io/projected/bbba48af-dcfc-4f28-a338-c5e6b09efc21-kube-api-access-d98b8\") pod \"maas-api-revoke-sub-9f2d70b7-9nd8k\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:26.791560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.790246    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:26.833394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.833265    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:26.855575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.855504    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:24:26.903493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.903445    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f2d70b7-9nd8k\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:26.938282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.938230    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d98b8\" (UniqueName: \"kubernetes.io/projected/bbba48af-dcfc-4f28-a338-c5e6b09efc21-kube-api-access-d98b8\") pod \"maas-api-revoke-sub-9f2d70b7-9nd8k\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:26.938605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.938580    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f2d70b7-9nd8k\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:27.031584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.919260    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.031584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:26.920133    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f2d70b7-9nd8k\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:27.031584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.001945    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f2d70b7-9nd8k\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:27.031584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.005866    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.047912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.047865    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.076049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.058965    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d98b8\" (UniqueName: \"kubernetes.io/projected/bbba48af-dcfc-4f28-a338-c5e6b09efc21-kube-api-access-d98b8\") pod \"maas-api-revoke-sub-9f2d70b7-9nd8k\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:27.262493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.262385    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.335084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.325819    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:27.414270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.414228    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:24:27.523090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.522938    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.567093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.566452    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 30 19:24:27.785681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.785636    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:27.785977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.785946    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.793337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.793295    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.794858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.794834    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:24:27.824090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.817394    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Oct 30 19:24:28.470320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.850742    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.511366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.870468    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.561082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:27.993781    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.561082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.001876    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.561082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.031155    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.561082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.031202    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded"
Oct 30 19:24:28.561082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.090276    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.586080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.115174    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:28.586080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.566349    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.586080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.310170    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.586080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.435870    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.586080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.300474    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.631133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.630292    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.844155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.815082    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.844889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.844837    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:28.845114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:28.845083    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.186321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.184701    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:29.186321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.184939    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.344550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.344496    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:29.344895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.344844    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.521279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.521165    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.570908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.570861    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.572416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.572382    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.598315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.598269    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.606235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.606174    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.606704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.606658    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.631340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.607916    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.631626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.630097    6777 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:29.631907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.630606    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.632304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.632270    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.824539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:29.824488    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.993089 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:24:29.985976    6777 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.853s"
Oct 30 19:24:30.002944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.002911    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:24:30.003287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.003267    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:24:30.003443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.003431    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:24:30.003555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.003541    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"]
Oct 30 19:24:30.134277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.134178    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf" podUID="d3d7cb45-4e6d-466b-aa7b-498e2a682c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.196259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.196227    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"]
Oct 30 19:24:30.196599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.196581    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:24:30.206019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.205989    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:30.350866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.350829    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:30.351198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.351166    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.392142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.372423    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdn8\" (UniqueName: \"kubernetes.io/projected/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-kube-api-access-kwdn8\") pod \"maas-api-revoke-sub-9a7cc615-5sd97\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:30.393445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.393407    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7cc615-5sd97\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:30.393876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.393853    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7cc615-5sd97\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:30.465084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.394320    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:30.465399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.465358    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.465559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.452099    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:24:30.465751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.465736    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:24:30.507556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.507227    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:24:30.507556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.507284    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:24:30.540612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.540566    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:30.540943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.540909    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.839368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.677935    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7cc615-5sd97\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:30.840336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.840299    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7cc615-5sd97\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:30.840857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.840833    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdn8\" (UniqueName: \"kubernetes.io/projected/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-kube-api-access-kwdn8\") pod \"maas-api-revoke-sub-9a7cc615-5sd97\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:30.842505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.678288    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.868083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.861178    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7cc615-5sd97\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:30.868083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.678391    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.868083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.678373    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.868083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.678330    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.870431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.678353    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.893582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.678424    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.894250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.678727    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.921084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.899158    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.921084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:30.760962    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7cc615-5sd97\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:31.034393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.034346    6777 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:31.034692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.034645    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.080681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.080641    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.172457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.172372    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdn8\" (UniqueName: \"kubernetes.io/projected/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-kube-api-access-kwdn8\") pod \"maas-api-revoke-sub-9a7cc615-5sd97\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:31.499287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.477333    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:31.607876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.558740    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 30 19:24:31.709561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.709514    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k" event={"ID":"bbba48af-dcfc-4f28-a338-c5e6b09efc21","Type":"ContainerStarted","Data":"6cc297d06a70383315418bd28465bf641248e54711f58a79beb6c6c5da9a1b48"}
Oct 30 19:24:31.846260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.846205    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.922343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.922310    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.922892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.922867    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.923373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.923351    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.924996 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.924976    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.925898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.925874    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.926317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.926295    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:24:32.018099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:31.929400    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:32.097226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.097127    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:32.104993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.104961    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"]
Oct 30 19:24:32.105276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.105263    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:24:32.204750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.180841    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:24:32.204750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.180923    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:24:32.204750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.180977    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:24:32.204750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.181024    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:24:32.204750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.199537    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:24:32.219493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.219469    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:24:32.219808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.219669    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:24:32.235243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.235194    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:24:32.605025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.604992    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"]
Oct 30 19:24:32.605408 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:32.605387    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4085ae2d_9fe7_4d3e_b5eb_7623baa7714e.slice/crio-b6a14f86973d07dfd812191aa3ecf8c7b001e6d950b79c5ae0585e2647a89fc1 WatchSource:0}: Error finding container b6a14f86973d07dfd812191aa3ecf8c7b001e6d950b79c5ae0585e2647a89fc1: Status 404 returned error can't find the container with id b6a14f86973d07dfd812191aa3ecf8c7b001e6d950b79c5ae0585e2647a89fc1
Oct 30 19:24:32.718944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.718901    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k" event={"ID":"bbba48af-dcfc-4f28-a338-c5e6b09efc21","Type":"ContainerStarted","Data":"ff755b7f47975f312b2c85cbb8209a19da23eab497814a6dc57869977ac7008f"}
Oct 30 19:24:32.720393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.720369    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97" event={"ID":"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e","Type":"ContainerStarted","Data":"b6a14f86973d07dfd812191aa3ecf8c7b001e6d950b79c5ae0585e2647a89fc1"}
Oct 30 19:24:32.741619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:32.741573    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k" podStartSLOduration=6.741560681 podStartE2EDuration="6.741560681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:32.737963746 +0000 UTC m=+1054.734879130" watchObservedRunningTime="2026-10-30 19:24:32.741560681 +0000 UTC m=+1054.738476105"
Oct 30 19:24:33.730456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:33.730420    6777 generic.go:356] "Generic (PLEG): container finished" podID="4085ae2d-9fe7-4d3e-b5eb-7623baa7714e" containerID="09a672bc9cb22dcd480d1b2774e14bcc434e777eb52f64a0c876bd3b9c905b3f" exitCode=0
Oct 30 19:24:33.731011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:33.730502    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97" event={"ID":"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e","Type":"ContainerDied","Data":"09a672bc9cb22dcd480d1b2774e14bcc434e777eb52f64a0c876bd3b9c905b3f"}
Oct 30 19:24:33.732249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:33.732224    6777 generic.go:356] "Generic (PLEG): container finished" podID="bbba48af-dcfc-4f28-a338-c5e6b09efc21" containerID="ff755b7f47975f312b2c85cbb8209a19da23eab497814a6dc57869977ac7008f" exitCode=0
Oct 30 19:24:33.732356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:33.732268    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k" event={"ID":"bbba48af-dcfc-4f28-a338-c5e6b09efc21","Type":"ContainerDied","Data":"ff755b7f47975f312b2c85cbb8209a19da23eab497814a6dc57869977ac7008f"}
Oct 30 19:24:34.952816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:34.952787    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:35.015877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.015806    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:35.043818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.043783    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-xks-ca\") pod \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") "
Oct 30 19:24:35.043998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.043829    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-openshift-ca\") pod \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") "
Oct 30 19:24:35.043998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.043889    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbba48af-dcfc-4f28-a338-c5e6b09efc21-kube-api-access-d98b8\" (UniqueName: \"kubernetes.io/projected/bbba48af-dcfc-4f28-a338-c5e6b09efc21-kube-api-access-d98b8\") pod \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") "
Oct 30 19:24:35.043998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.043930    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-kube-api-access-kwdn8\" (UniqueName: \"kubernetes.io/projected/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-kube-api-access-kwdn8\") pod \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") "
Oct 30 19:24:35.044239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.044051    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-xks-ca\") pod \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\" (UID: \"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e\") "
Oct 30 19:24:35.044239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.044102    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-openshift-ca\") pod \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\" (UID: \"bbba48af-dcfc-4f28-a338-c5e6b09efc21\") "
Oct 30 19:24:35.044345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.044288    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-openshift-ca" pod "4085ae2d-9fe7-4d3e-b5eb-7623baa7714e" (UID: "4085ae2d-9fe7-4d3e-b5eb-7623baa7714e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:35.044695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.044660    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-openshift-ca" pod "bbba48af-dcfc-4f28-a338-c5e6b09efc21" (UID: "bbba48af-dcfc-4f28-a338-c5e6b09efc21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:35.044875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.044854    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:35.044934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.044882    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:35.048080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.046690    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbba48af-dcfc-4f28-a338-c5e6b09efc21-kube-api-access-d98b8" pod "bbba48af-dcfc-4f28-a338-c5e6b09efc21" (UID: "bbba48af-dcfc-4f28-a338-c5e6b09efc21"). InnerVolumeSpecName "kube-api-access-d98b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:35.048080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.046810    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-xks-ca" pod "bbba48af-dcfc-4f28-a338-c5e6b09efc21" (UID: "bbba48af-dcfc-4f28-a338-c5e6b09efc21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:35.048080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.047138    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-xks-ca" pod "4085ae2d-9fe7-4d3e-b5eb-7623baa7714e" (UID: "4085ae2d-9fe7-4d3e-b5eb-7623baa7714e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:35.068294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.066666    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-kube-api-access-kwdn8" pod "4085ae2d-9fe7-4d3e-b5eb-7623baa7714e" (UID: "4085ae2d-9fe7-4d3e-b5eb-7623baa7714e"). InnerVolumeSpecName "kube-api-access-kwdn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:35.146895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.146859    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:35.146895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.146885    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbba48af-dcfc-4f28-a338-c5e6b09efc21-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:35.146895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.146896    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d98b8\" (UniqueName: \"kubernetes.io/projected/bbba48af-dcfc-4f28-a338-c5e6b09efc21-kube-api-access-d98b8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:35.147257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.146912    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwdn8\" (UniqueName: \"kubernetes.io/projected/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e-kube-api-access-kwdn8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:35.760743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.760703    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97" event={"ID":"4085ae2d-9fe7-4d3e-b5eb-7623baa7714e","Type":"ContainerDied","Data":"b6a14f86973d07dfd812191aa3ecf8c7b001e6d950b79c5ae0585e2647a89fc1"}
Oct 30 19:24:35.761017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.760768    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6a14f86973d07dfd812191aa3ecf8c7b001e6d950b79c5ae0585e2647a89fc1"
Oct 30 19:24:35.761017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.760850    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"
Oct 30 19:24:35.768250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.768194    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k" event={"ID":"bbba48af-dcfc-4f28-a338-c5e6b09efc21","Type":"ContainerDied","Data":"6cc297d06a70383315418bd28465bf641248e54711f58a79beb6c6c5da9a1b48"}
Oct 30 19:24:35.768450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.768274    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cc297d06a70383315418bd28465bf641248e54711f58a79beb6c6c5da9a1b48"
Oct 30 19:24:35.768450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:35.768352    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"
Oct 30 19:24:43.642923 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:24:43.642883    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:24:43.785721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.785683    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"]
Oct 30 19:24:43.786403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.786386    6777 cpu_manager.go:414] "containerMap: removing container" podUID="bbba48af-dcfc-4f28-a338-c5e6b09efc21" containerName="revoke-keys"
Oct 30 19:24:43.786464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.786404    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbba48af-dcfc-4f28-a338-c5e6b09efc21" containerName="revoke-keys"
Oct 30 19:24:43.786464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.786446    6777 cpu_manager.go:414] "containerMap: removing container" podUID="4085ae2d-9fe7-4d3e-b5eb-7623baa7714e" containerName="revoke-keys"
Oct 30 19:24:43.786464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.786451    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4085ae2d-9fe7-4d3e-b5eb-7623baa7714e" containerName="revoke-keys"
Oct 30 19:24:43.786555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.786540    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbba48af-dcfc-4f28-a338-c5e6b09efc21" containerName="revoke-keys"
Oct 30 19:24:43.786593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.786555    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="4085ae2d-9fe7-4d3e-b5eb-7623baa7714e" containerName="revoke-keys"
Oct 30 19:24:43.791574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.791554    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:43.799316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.799288    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"]
Oct 30 19:24:43.878036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.878005    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:43.878347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.878325    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:24:43.890020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.889988    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac91a75-bcmz5\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:43.890020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.890022    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tbtp\" (UniqueName: \"kubernetes.io/projected/93692622-047d-43d5-a55c-9fa4d5d718b6-kube-api-access-7tbtp\") pod \"maas-api-revoke-sub-dac91a75-bcmz5\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:43.890270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.890249    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac91a75-bcmz5\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:43.991473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.991389    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac91a75-bcmz5\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:43.991642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.991527    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac91a75-bcmz5\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:43.991642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.991549    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tbtp\" (UniqueName: \"kubernetes.io/projected/93692622-047d-43d5-a55c-9fa4d5d718b6-kube-api-access-7tbtp\") pod \"maas-api-revoke-sub-dac91a75-bcmz5\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:43.992251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.992231    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac91a75-bcmz5\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:43.993968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:43.993945    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac91a75-bcmz5\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:44.009707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.009673    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tbtp\" (UniqueName: \"kubernetes.io/projected/93692622-047d-43d5-a55c-9fa4d5d718b6-kube-api-access-7tbtp\") pod \"maas-api-revoke-sub-dac91a75-bcmz5\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:44.110480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.110434    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:44.345573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.345539    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"]
Oct 30 19:24:44.354669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.354631    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.363846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.363815    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7c74535f-openshift-default-dockercfg-t94qq\""
Oct 30 19:24:44.372308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.372275    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"]
Oct 30 19:24:44.424126 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:44.424088    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93692622_047d_43d5_a55c_9fa4d5d718b6.slice/crio-3c4e45a056e3d0e3ccf96b14939a358ac70dff851d6ae3155ab395213486c2fe WatchSource:0}: Error finding container 3c4e45a056e3d0e3ccf96b14939a358ac70dff851d6ae3155ab395213486c2fe: Status 404 returned error can't find the container with id 3c4e45a056e3d0e3ccf96b14939a358ac70dff851d6ae3155ab395213486c2fe
Oct 30 19:24:44.424849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.424825    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"]
Oct 30 19:24:44.516491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.516454    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f418d9df-8c20-45bc-913b-d63999cf42a1-istiod-ca-cert\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.516666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.516498    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-token\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.516666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.516620    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-envoy\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.516787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.516773    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-credential-socket\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.516823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.516811    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-podinfo\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.516863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.516831    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-certs\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.516967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.516944    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-socket\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.517035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.516979    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-data\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.517035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.517007    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjl59\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-kube-api-access-zjl59\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.619475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.619360    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-envoy\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.619627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.619611    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-credential-socket\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.619689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.619663    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-podinfo\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.619730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.619707    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-certs\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.622879    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-certs\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.619743    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-socket\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.623131    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-data\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.623180    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjl59\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-kube-api-access-zjl59\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.623217    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-credential-socket\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.623241    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-socket\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.623446    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f418d9df-8c20-45bc-913b-d63999cf42a1-istiod-ca-cert\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.623453    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-data\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.623506    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-token\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.624713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.624136    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f418d9df-8c20-45bc-913b-d63999cf42a1-istiod-ca-cert\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.649935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.649900    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-podinfo\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.652144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.652049    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-envoy\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.652371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.652351    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-token\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.652721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.652700    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjl59\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-kube-api-access-zjl59\") pod \"e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") " pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.677197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.677168    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:44.864614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.864581    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5" event={"ID":"93692622-047d-43d5-a55c-9fa4d5d718b6","Type":"ContainerStarted","Data":"3c4e45a056e3d0e3ccf96b14939a358ac70dff851d6ae3155ab395213486c2fe"}
Oct 30 19:24:44.994110 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:44.994073    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf418d9df_8c20_45bc_913b_d63999cf42a1.slice/crio-8d8bf0477a2a3204f65ce2180b31482ccdb9d3722e5bdef41c8e8f3498730f8f WatchSource:0}: Error finding container 8d8bf0477a2a3204f65ce2180b31482ccdb9d3722e5bdef41c8e8f3498730f8f: Status 404 returned error can't find the container with id 8d8bf0477a2a3204f65ce2180b31482ccdb9d3722e5bdef41c8e8f3498730f8f
Oct 30 19:24:44.994442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.994420    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"]
Oct 30 19:24:44.997415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.997378    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:24:44.997523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.997452    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:24:44.997523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:44.997503    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:24:45.910281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:45.903591    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp" event={"ID":"f418d9df-8c20-45bc-913b-d63999cf42a1","Type":"ContainerStarted","Data":"510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850"}
Oct 30 19:24:45.910281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:45.903664    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp" event={"ID":"f418d9df-8c20-45bc-913b-d63999cf42a1","Type":"ContainerStarted","Data":"8d8bf0477a2a3204f65ce2180b31482ccdb9d3722e5bdef41c8e8f3498730f8f"}
Oct 30 19:24:45.926402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:45.925714    6777 generic.go:356] "Generic (PLEG): container finished" podID="93692622-047d-43d5-a55c-9fa4d5d718b6" containerID="2f831fc502aacdce1e9681f9f5881052a1babfe4be76c6d29a355153de34e5fc" exitCode=0
Oct 30 19:24:45.926402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:45.925804    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5" event={"ID":"93692622-047d-43d5-a55c-9fa4d5d718b6","Type":"ContainerDied","Data":"2f831fc502aacdce1e9681f9f5881052a1babfe4be76c6d29a355153de34e5fc"}
Oct 30 19:24:45.955126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:45.946489    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp" podStartSLOduration=1.946470743 podStartE2EDuration="1.946470743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:45.944402518 +0000 UTC m=+1067.941317956" watchObservedRunningTime="2026-10-30 19:24:45.946470743 +0000 UTC m=+1067.943386129"
Oct 30 19:24:46.678373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:46.678341    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:46.684021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:46.683995    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:46.935774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:46.935247    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:46.936569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:46.936546    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:24:47.143075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.143039    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:47.213835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.213750    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-xks-ca\") pod \"93692622-047d-43d5-a55c-9fa4d5d718b6\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") "
Oct 30 19:24:47.214034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.213871    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-openshift-ca\") pod \"93692622-047d-43d5-a55c-9fa4d5d718b6\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") "
Oct 30 19:24:47.214328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.214295    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-openshift-ca" pod "93692622-047d-43d5-a55c-9fa4d5d718b6" (UID: "93692622-047d-43d5-a55c-9fa4d5d718b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:47.214452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.214435    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93692622-047d-43d5-a55c-9fa4d5d718b6-kube-api-access-7tbtp\" (UniqueName: \"kubernetes.io/projected/93692622-047d-43d5-a55c-9fa4d5d718b6-kube-api-access-7tbtp\") pod \"93692622-047d-43d5-a55c-9fa4d5d718b6\" (UID: \"93692622-047d-43d5-a55c-9fa4d5d718b6\") "
Oct 30 19:24:47.216493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.216470    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:47.217226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.217195    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93692622-047d-43d5-a55c-9fa4d5d718b6-kube-api-access-7tbtp" pod "93692622-047d-43d5-a55c-9fa4d5d718b6" (UID: "93692622-047d-43d5-a55c-9fa4d5d718b6"). InnerVolumeSpecName "kube-api-access-7tbtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:47.217435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.217406    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-xks-ca" pod "93692622-047d-43d5-a55c-9fa4d5d718b6" (UID: "93692622-047d-43d5-a55c-9fa4d5d718b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:47.318798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.318762    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tbtp\" (UniqueName: \"kubernetes.io/projected/93692622-047d-43d5-a55c-9fa4d5d718b6-kube-api-access-7tbtp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:47.318798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.318798    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93692622-047d-43d5-a55c-9fa4d5d718b6-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:47.395957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.395918    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"]
Oct 30 19:24:47.396970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.396938    6777 cpu_manager.go:414] "containerMap: removing container" podUID="93692622-047d-43d5-a55c-9fa4d5d718b6" containerName="revoke-keys"
Oct 30 19:24:47.396970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.396964    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93692622-047d-43d5-a55c-9fa4d5d718b6" containerName="revoke-keys"
Oct 30 19:24:47.397177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.397145    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="93692622-047d-43d5-a55c-9fa4d5d718b6" containerName="revoke-keys"
Oct 30 19:24:47.402325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.402293    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.414080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.410836    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"]
Oct 30 19:24:47.548841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.548793    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-250fb057-c6n8m\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.549015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.548975    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pghth\" (UniqueName: \"kubernetes.io/projected/7e29868f-1ddd-4eaa-b555-872af2e0cf72-kube-api-access-pghth\") pod \"maas-api-revoke-keys-e2e-ait-250fb057-c6n8m\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.549162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.549134    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-250fb057-c6n8m\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.650986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.650941    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-250fb057-c6n8m\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.651200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.651122    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-250fb057-c6n8m\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.651269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.651255    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pghth\" (UniqueName: \"kubernetes.io/projected/7e29868f-1ddd-4eaa-b555-872af2e0cf72-kube-api-access-pghth\") pod \"maas-api-revoke-keys-e2e-ait-250fb057-c6n8m\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.651834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.651803    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-250fb057-c6n8m\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.653463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.653441    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-250fb057-c6n8m\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.663909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.663883    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pghth\" (UniqueName: \"kubernetes.io/projected/7e29868f-1ddd-4eaa-b555-872af2e0cf72-kube-api-access-pghth\") pod \"maas-api-revoke-keys-e2e-ait-250fb057-c6n8m\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.752513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.752479    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:47.945483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.945448    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5" event={"ID":"93692622-047d-43d5-a55c-9fa4d5d718b6","Type":"ContainerDied","Data":"3c4e45a056e3d0e3ccf96b14939a358ac70dff851d6ae3155ab395213486c2fe"}
Oct 30 19:24:47.945874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.945530    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c4e45a056e3d0e3ccf96b14939a358ac70dff851d6ae3155ab395213486c2fe"
Oct 30 19:24:47.945874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:47.945466    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"
Oct 30 19:24:48.058086 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:48.058044    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e29868f_1ddd_4eaa_b555_872af2e0cf72.slice/crio-c8be014b5d83e5e227b9ec9826885af85b618817f59b54162ffd9e6c7ef37cd0 WatchSource:0}: Error finding container c8be014b5d83e5e227b9ec9826885af85b618817f59b54162ffd9e6c7ef37cd0: Status 404 returned error can't find the container with id c8be014b5d83e5e227b9ec9826885af85b618817f59b54162ffd9e6c7ef37cd0
Oct 30 19:24:48.059999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:48.059933    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"]
Oct 30 19:24:48.964103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:48.960949    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m" event={"ID":"7e29868f-1ddd-4eaa-b555-872af2e0cf72","Type":"ContainerStarted","Data":"a6053eff2e797b6b36b1f3aa6a4066bcb40c32b2046426b4926d10e81dd90c09"}
Oct 30 19:24:48.964103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:48.961038    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m" event={"ID":"7e29868f-1ddd-4eaa-b555-872af2e0cf72","Type":"ContainerStarted","Data":"c8be014b5d83e5e227b9ec9826885af85b618817f59b54162ffd9e6c7ef37cd0"}
Oct 30 19:24:48.989867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:48.989820    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m" podStartSLOduration=1.98980609 podStartE2EDuration="1.98980609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:48.987796629 +0000 UTC m=+1070.984712016" watchObservedRunningTime="2026-10-30 19:24:48.98980609 +0000 UTC m=+1070.986721476"
Oct 30 19:24:49.075684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.075641    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"]
Oct 30 19:24:49.090455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.090427    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:24:49.092011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.091975    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"]
Oct 30 19:24:49.095361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.095338    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7c74535f-dockercfg-4tfmz\""
Oct 30 19:24:49.165448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.165417    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7c74535f\""
Oct 30 19:24:49.205954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.204524    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdr5v\" (UniqueName: \"kubernetes.io/projected/64cdf9ed-f694-4a32-89a1-9d9d20009076-kube-api-access-bdr5v\") pod \"payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4\" (UID: \"64cdf9ed-f694-4a32-89a1-9d9d20009076\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:24:49.205954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.204637    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cdf9ed-f694-4a32-89a1-9d9d20009076-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4\" (UID: \"64cdf9ed-f694-4a32-89a1-9d9d20009076\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:24:49.310129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.307540    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdr5v\" (UniqueName: \"kubernetes.io/projected/64cdf9ed-f694-4a32-89a1-9d9d20009076-kube-api-access-bdr5v\") pod \"payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4\" (UID: \"64cdf9ed-f694-4a32-89a1-9d9d20009076\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:24:49.310129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.307625    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cdf9ed-f694-4a32-89a1-9d9d20009076-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4\" (UID: \"64cdf9ed-f694-4a32-89a1-9d9d20009076\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:24:49.310129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.309156    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cdf9ed-f694-4a32-89a1-9d9d20009076-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4\" (UID: \"64cdf9ed-f694-4a32-89a1-9d9d20009076\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:24:49.334863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.334835    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdr5v\" (UniqueName: \"kubernetes.io/projected/64cdf9ed-f694-4a32-89a1-9d9d20009076-kube-api-access-bdr5v\") pod \"payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4\" (UID: \"64cdf9ed-f694-4a32-89a1-9d9d20009076\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:24:49.444484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.444448    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:24:49.750139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.750101    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"]
Oct 30 19:24:49.750430 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:49.750403    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cdf9ed_f694_4a32_89a1_9d9d20009076.slice/crio-01f7eb13e0abbe1549c326f190acf715c5cf5b8efa6781ee93a44bd69ff30bc9 WatchSource:0}: Error finding container 01f7eb13e0abbe1549c326f190acf715c5cf5b8efa6781ee93a44bd69ff30bc9: Status 404 returned error can't find the container with id 01f7eb13e0abbe1549c326f190acf715c5cf5b8efa6781ee93a44bd69ff30bc9
Oct 30 19:24:49.970936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.970901    6777 generic.go:356] "Generic (PLEG): container finished" podID="7e29868f-1ddd-4eaa-b555-872af2e0cf72" containerID="a6053eff2e797b6b36b1f3aa6a4066bcb40c32b2046426b4926d10e81dd90c09" exitCode=0
Oct 30 19:24:49.971427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.970940    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m" event={"ID":"7e29868f-1ddd-4eaa-b555-872af2e0cf72","Type":"ContainerDied","Data":"a6053eff2e797b6b36b1f3aa6a4066bcb40c32b2046426b4926d10e81dd90c09"}
Oct 30 19:24:49.972810 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.972783    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4" event={"ID":"64cdf9ed-f694-4a32-89a1-9d9d20009076","Type":"ContainerStarted","Data":"b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34"}
Oct 30 19:24:49.972952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.972831    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4" event={"ID":"64cdf9ed-f694-4a32-89a1-9d9d20009076","Type":"ContainerStarted","Data":"01f7eb13e0abbe1549c326f190acf715c5cf5b8efa6781ee93a44bd69ff30bc9"}
Oct 30 19:24:49.972952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:49.972904    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:24:50.023485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:50.023366    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4" podStartSLOduration=1.023352138 podStartE2EDuration="1.023352138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:50.020813058 +0000 UTC m=+1072.017728443" watchObservedRunningTime="2026-10-30 19:24:50.023352138 +0000 UTC m=+1072.020267523"
Oct 30 19:24:51.169927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.169892    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:51.261405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.261315    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-openshift-ca\") pod \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") "
Oct 30 19:24:51.261405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.261375    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-xks-ca\") pod \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") "
Oct 30 19:24:51.261646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.261438    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e29868f-1ddd-4eaa-b555-872af2e0cf72-kube-api-access-pghth\" (UniqueName: \"kubernetes.io/projected/7e29868f-1ddd-4eaa-b555-872af2e0cf72-kube-api-access-pghth\") pod \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\" (UID: \"7e29868f-1ddd-4eaa-b555-872af2e0cf72\") "
Oct 30 19:24:51.261743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.261714    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-openshift-ca" pod "7e29868f-1ddd-4eaa-b555-872af2e0cf72" (UID: "7e29868f-1ddd-4eaa-b555-872af2e0cf72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:51.262166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.262144    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:51.263585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.263564    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-xks-ca" pod "7e29868f-1ddd-4eaa-b555-872af2e0cf72" (UID: "7e29868f-1ddd-4eaa-b555-872af2e0cf72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:51.263676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.263660    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e29868f-1ddd-4eaa-b555-872af2e0cf72-kube-api-access-pghth" pod "7e29868f-1ddd-4eaa-b555-872af2e0cf72" (UID: "7e29868f-1ddd-4eaa-b555-872af2e0cf72"). InnerVolumeSpecName "kube-api-access-pghth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:51.364248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.364211    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29868f-1ddd-4eaa-b555-872af2e0cf72-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:51.364248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.364240    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pghth\" (UniqueName: \"kubernetes.io/projected/7e29868f-1ddd-4eaa-b555-872af2e0cf72-kube-api-access-pghth\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:51.470881 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.470848    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:24:51.993245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.993217    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"
Oct 30 19:24:51.993396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.993248    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m" event={"ID":"7e29868f-1ddd-4eaa-b555-872af2e0cf72","Type":"ContainerDied","Data":"c8be014b5d83e5e227b9ec9826885af85b618817f59b54162ffd9e6c7ef37cd0"}
Oct 30 19:24:51.993396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:51.993303    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8be014b5d83e5e227b9ec9826885af85b618817f59b54162ffd9e6c7ef37cd0"
Oct 30 19:24:53.884085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:53.877144    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.062083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.056778    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:54.062083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.056869    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.091086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.088704    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.164141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.158148    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:24:54.877080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.857148    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.877080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.869360    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:24:54.881083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.879339    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.881083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.879680    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.881083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.880636    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:24:54.906081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.902126    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.951084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:54.937071    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.276162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.242180    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:24:55.276162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.243338    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:24:55.276162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.252598    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:55.276162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.253463    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:24:55.380860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.380661    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:24:55.432246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.429352    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"]
Oct 30 19:24:55.432246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.430417    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7e29868f-1ddd-4eaa-b555-872af2e0cf72" containerName="revoke-keys"
Oct 30 19:24:55.432246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.430440    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e29868f-1ddd-4eaa-b555-872af2e0cf72" containerName="revoke-keys"
Oct 30 19:24:55.432246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.430624    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e29868f-1ddd-4eaa-b555-872af2e0cf72" containerName="revoke-keys"
Oct 30 19:24:55.439234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.439209    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:55.453976 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.453934    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:55.454298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.454280    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:24:55.498236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.498197    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"]
Oct 30 19:24:55.562923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.560734    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:55.562923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.560842    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6k97\" (UniqueName: \"kubernetes.io/projected/a67900cd-a131-425b-9a3b-a86a507bacfb-kube-api-access-v6k97\") pod \"maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:55.562923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.561045    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:55.623092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.622651    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:24:55.718079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.704877    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:55.718079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.704981    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6k97\" (UniqueName: \"kubernetes.io/projected/a67900cd-a131-425b-9a3b-a86a507bacfb-kube-api-access-v6k97\") pod \"maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:55.718079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.705270    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:55.718079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.706140    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:55.718079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.709632    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:24:55.733176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.728844    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:55.785889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:55.785816    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6k97\" (UniqueName: \"kubernetes.io/projected/a67900cd-a131-425b-9a3b-a86a507bacfb-kube-api-access-v6k97\") pod \"maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:56.078552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:56.078515    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:56.381130 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:56.381094    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67900cd_a131_425b_9a3b_a86a507bacfb.slice/crio-8affbdad1cc400a68cb9f8ae36ae7b4f352f3d2ef37a9e03c17ac44ca836a6ba WatchSource:0}: Error finding container 8affbdad1cc400a68cb9f8ae36ae7b4f352f3d2ef37a9e03c17ac44ca836a6ba: Status 404 returned error can't find the container with id 8affbdad1cc400a68cb9f8ae36ae7b4f352f3d2ef37a9e03c17ac44ca836a6ba
Oct 30 19:24:56.382266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:56.382239    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"]
Oct 30 19:24:56.575489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:56.575449    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8" event={"ID":"a67900cd-a131-425b-9a3b-a86a507bacfb","Type":"ContainerStarted","Data":"8affbdad1cc400a68cb9f8ae36ae7b4f352f3d2ef37a9e03c17ac44ca836a6ba"}
Oct 30 19:24:57.592050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:57.592014    6777 generic.go:356] "Generic (PLEG): container finished" podID="a67900cd-a131-425b-9a3b-a86a507bacfb" containerID="899b729e552acbbc8f1e9fc7fe3f6324163d92a9aba3306fbb0d8ef2290ebd6d" exitCode=0
Oct 30 19:24:57.592550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:57.592073    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8" event={"ID":"a67900cd-a131-425b-9a3b-a86a507bacfb","Type":"ContainerDied","Data":"899b729e552acbbc8f1e9fc7fe3f6324163d92a9aba3306fbb0d8ef2290ebd6d"}
Oct 30 19:24:58.824298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.824251    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:58.872227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.872127    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-xks-ca\") pod \"a67900cd-a131-425b-9a3b-a86a507bacfb\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") "
Oct 30 19:24:58.872394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.872239    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67900cd-a131-425b-9a3b-a86a507bacfb-kube-api-access-v6k97\" (UniqueName: \"kubernetes.io/projected/a67900cd-a131-425b-9a3b-a86a507bacfb-kube-api-access-v6k97\") pod \"a67900cd-a131-425b-9a3b-a86a507bacfb\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") "
Oct 30 19:24:58.872478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.872400    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-openshift-ca\") pod \"a67900cd-a131-425b-9a3b-a86a507bacfb\" (UID: \"a67900cd-a131-425b-9a3b-a86a507bacfb\") "
Oct 30 19:24:58.872955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.872925    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-openshift-ca" pod "a67900cd-a131-425b-9a3b-a86a507bacfb" (UID: "a67900cd-a131-425b-9a3b-a86a507bacfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:58.873281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.873259    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:58.874960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.874924    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-xks-ca" pod "a67900cd-a131-425b-9a3b-a86a507bacfb" (UID: "a67900cd-a131-425b-9a3b-a86a507bacfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:58.876427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.876402    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67900cd-a131-425b-9a3b-a86a507bacfb-kube-api-access-v6k97" pod "a67900cd-a131-425b-9a3b-a86a507bacfb" (UID: "a67900cd-a131-425b-9a3b-a86a507bacfb"). InnerVolumeSpecName "kube-api-access-v6k97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:58.975024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.974977    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67900cd-a131-425b-9a3b-a86a507bacfb-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:58.975024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.975021    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6k97\" (UniqueName: \"kubernetes.io/projected/a67900cd-a131-425b-9a3b-a86a507bacfb-kube-api-access-v6k97\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:58.987748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.987714    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"]
Oct 30 19:24:58.988544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.988527    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a67900cd-a131-425b-9a3b-a86a507bacfb" containerName="revoke-keys"
Oct 30 19:24:58.988615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.988545    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67900cd-a131-425b-9a3b-a86a507bacfb" containerName="revoke-keys"
Oct 30 19:24:58.988693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.988684    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67900cd-a131-425b-9a3b-a86a507bacfb" containerName="revoke-keys"
Oct 30 19:24:58.992639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:58.992620    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.003398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.003264    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"]
Oct 30 19:24:59.078480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.078052    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5lnc\" (UniqueName: \"kubernetes.io/projected/0417cf65-b16e-46c2-b2c5-396fee491051-kube-api-access-s5lnc\") pod \"maas-api-revoke-sub-3ee94ead-brtdt\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.078480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.078361    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee94ead-brtdt\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.078480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.078446    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee94ead-brtdt\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.180790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.180698    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee94ead-brtdt\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.180945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.180819    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lnc\" (UniqueName: \"kubernetes.io/projected/0417cf65-b16e-46c2-b2c5-396fee491051-kube-api-access-s5lnc\") pod \"maas-api-revoke-sub-3ee94ead-brtdt\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.181089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.181047    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee94ead-brtdt\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.181835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.181806    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee94ead-brtdt\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.183344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.183320    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee94ead-brtdt\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.195133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.195108    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5lnc\" (UniqueName: \"kubernetes.io/projected/0417cf65-b16e-46c2-b2c5-396fee491051-kube-api-access-s5lnc\") pod \"maas-api-revoke-sub-3ee94ead-brtdt\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.317052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.317025    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:24:59.605849 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:24:59.605820    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0417cf65_b16e_46c2_b2c5_396fee491051.slice/crio-59510204c450a0816a96e9afeac8f578c8b9f98cc24ad31f736a57a8c6a2887c WatchSource:0}: Error finding container 59510204c450a0816a96e9afeac8f578c8b9f98cc24ad31f736a57a8c6a2887c: Status 404 returned error can't find the container with id 59510204c450a0816a96e9afeac8f578c8b9f98cc24ad31f736a57a8c6a2887c
Oct 30 19:24:59.607400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.607369    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"]
Oct 30 19:24:59.619953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.615306    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"
Oct 30 19:24:59.619953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.615364    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8" event={"ID":"a67900cd-a131-425b-9a3b-a86a507bacfb","Type":"ContainerDied","Data":"8affbdad1cc400a68cb9f8ae36ae7b4f352f3d2ef37a9e03c17ac44ca836a6ba"}
Oct 30 19:24:59.619953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:24:59.615420    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8affbdad1cc400a68cb9f8ae36ae7b4f352f3d2ef37a9e03c17ac44ca836a6ba"
Oct 30 19:25:00.640710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:00.640675    6777 generic.go:356] "Generic (PLEG): container finished" podID="0417cf65-b16e-46c2-b2c5-396fee491051" containerID="99b4c0fa8866d3a0d8fd132bd8444c36eb1ced92d96c9d9c974cab2b17cb8f50" exitCode=0
Oct 30 19:25:00.641144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:00.640754    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt" event={"ID":"0417cf65-b16e-46c2-b2c5-396fee491051","Type":"ContainerDied","Data":"99b4c0fa8866d3a0d8fd132bd8444c36eb1ced92d96c9d9c974cab2b17cb8f50"}
Oct 30 19:25:00.641144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:00.640826    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt" event={"ID":"0417cf65-b16e-46c2-b2c5-396fee491051","Type":"ContainerStarted","Data":"59510204c450a0816a96e9afeac8f578c8b9f98cc24ad31f736a57a8c6a2887c"}
Oct 30 19:25:00.984756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:00.984669    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:25:01.828482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:01.828452    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:25:01.965322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:01.965237    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0417cf65-b16e-46c2-b2c5-396fee491051-kube-api-access-s5lnc\" (UniqueName: \"kubernetes.io/projected/0417cf65-b16e-46c2-b2c5-396fee491051-kube-api-access-s5lnc\") pod \"0417cf65-b16e-46c2-b2c5-396fee491051\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") "
Oct 30 19:25:01.965322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:01.965280    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-xks-ca\") pod \"0417cf65-b16e-46c2-b2c5-396fee491051\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") "
Oct 30 19:25:01.965545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:01.965400    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-openshift-ca\") pod \"0417cf65-b16e-46c2-b2c5-396fee491051\" (UID: \"0417cf65-b16e-46c2-b2c5-396fee491051\") "
Oct 30 19:25:01.965721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:01.965699    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-openshift-ca" pod "0417cf65-b16e-46c2-b2c5-396fee491051" (UID: "0417cf65-b16e-46c2-b2c5-396fee491051"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:01.966233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:01.966212    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:01.967444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:01.967411    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0417cf65-b16e-46c2-b2c5-396fee491051-kube-api-access-s5lnc" pod "0417cf65-b16e-46c2-b2c5-396fee491051" (UID: "0417cf65-b16e-46c2-b2c5-396fee491051"). InnerVolumeSpecName "kube-api-access-s5lnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:01.967444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:01.967413    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-xks-ca" pod "0417cf65-b16e-46c2-b2c5-396fee491051" (UID: "0417cf65-b16e-46c2-b2c5-396fee491051"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:02.068402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:02.068366    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5lnc\" (UniqueName: \"kubernetes.io/projected/0417cf65-b16e-46c2-b2c5-396fee491051-kube-api-access-s5lnc\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:02.068402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:02.068400    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0417cf65-b16e-46c2-b2c5-396fee491051-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:02.659762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:02.659727    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt" event={"ID":"0417cf65-b16e-46c2-b2c5-396fee491051","Type":"ContainerDied","Data":"59510204c450a0816a96e9afeac8f578c8b9f98cc24ad31f736a57a8c6a2887c"}
Oct 30 19:25:02.659762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:02.659745    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"
Oct 30 19:25:02.660005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:02.659777    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59510204c450a0816a96e9afeac8f578c8b9f98cc24ad31f736a57a8c6a2887c"
Oct 30 19:25:06.856545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:06.856512    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"]
Oct 30 19:25:06.861207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:06.861174    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-250fb057-c6n8m"]
Oct 30 19:25:08.115176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.115143    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e29868f-1ddd-4eaa-b555-872af2e0cf72" path="/var/lib/kubelet/pods/7e29868f-1ddd-4eaa-b555-872af2e0cf72/volumes"
Oct 30 19:25:08.152153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.152113    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"]
Oct 30 19:25:08.152924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.152904    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0417cf65-b16e-46c2-b2c5-396fee491051" containerName="revoke-keys"
Oct 30 19:25:08.152924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.152922    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0417cf65-b16e-46c2-b2c5-396fee491051" containerName="revoke-keys"
Oct 30 19:25:08.153135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.153097    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0417cf65-b16e-46c2-b2c5-396fee491051" containerName="revoke-keys"
Oct 30 19:25:08.158323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.158302    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.166009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.165974    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c5ed0670-openshift-default-dockercfg-bmwnc\""
Oct 30 19:25:08.170252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.170220    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"]
Oct 30 19:25:08.171435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.171408    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-socket\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.171565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.171482    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-envoy\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.171565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.171545    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-istio-token\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.171672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.171586    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-credential-socket\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.171821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.171797    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-data\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.171925 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.171908    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefcb615-55d2-426c-b13c-f83582f841d2-istio-podinfo\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.172012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.171987    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm44g\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-kube-api-access-zm44g\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.172067 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.172042    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-certs\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.172257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.172223    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefcb615-55d2-426c-b13c-f83582f841d2-istiod-ca-cert\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.275485    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-data\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.275602    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefcb615-55d2-426c-b13c-f83582f841d2-istio-podinfo\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.275630    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm44g\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-kube-api-access-zm44g\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.275670    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-certs\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.275772    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefcb615-55d2-426c-b13c-f83582f841d2-istiod-ca-cert\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.275831    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-socket\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.275908    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-envoy\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.275939    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-istio-token\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.275987    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-credential-socket\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.282081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.276622    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-credential-socket\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.325109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.314619    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-socket\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.325109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.317816    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefcb615-55d2-426c-b13c-f83582f841d2-istiod-ca-cert\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.325109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.318221    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-certs\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.325109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.321183    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-data\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.347351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.347245    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-istio-token\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.371143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.360403    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-envoy\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.371143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.360986    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefcb615-55d2-426c-b13c-f83582f841d2-istio-podinfo\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.371143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.363797    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm44g\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-kube-api-access-zm44g\") pod \"e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") " pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.476993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.476907    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:08.790127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.790092    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"]
Oct 30 19:25:08.794194 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:25:08.793797    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefcb615_55d2_426c_b13c_f83582f841d2.slice/crio-5e94376b55e0aaf20e998bf3c8d231f225fa43e85d0636b2df49d6bc4b143dae WatchSource:0}: Error finding container 5e94376b55e0aaf20e998bf3c8d231f225fa43e85d0636b2df49d6bc4b143dae: Status 404 returned error can't find the container with id 5e94376b55e0aaf20e998bf3c8d231f225fa43e85d0636b2df49d6bc4b143dae
Oct 30 19:25:08.796709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.796669    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:08.796863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.796745    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:08.796863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:08.796801    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:09.735991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:09.735941    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb" event={"ID":"eefcb615-55d2-426c-b13c-f83582f841d2","Type":"ContainerStarted","Data":"643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704"}
Oct 30 19:25:09.736442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:09.736098    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb" event={"ID":"eefcb615-55d2-426c-b13c-f83582f841d2","Type":"ContainerStarted","Data":"5e94376b55e0aaf20e998bf3c8d231f225fa43e85d0636b2df49d6bc4b143dae"}
Oct 30 19:25:09.763495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:09.763440    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb" podStartSLOduration=1.7634254089999999 podStartE2EDuration="1.763425409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:09.761766727 +0000 UTC m=+1091.758682112" watchObservedRunningTime="2026-10-30 19:25:09.763425409 +0000 UTC m=+1091.760340793"
Oct 30 19:25:10.477568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:10.477526    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:10.483806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:10.483776    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:10.771172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:10.766340    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:10.771172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:10.767696    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:12.039147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:12.035540    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"]
Oct 30 19:25:12.051134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:12.051076    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e308e736-sdtc8"]
Oct 30 19:25:12.115951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:12.115913    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67900cd-a131-425b-9a3b-a86a507bacfb" path="/var/lib/kubelet/pods/a67900cd-a131-425b-9a3b-a86a507bacfb/volumes"
Oct 30 19:25:13.650612 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:25:13.650571    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:25:14.392414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.392004    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"]
Oct 30 19:25:14.398122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.398051    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.403565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.403532    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"]
Oct 30 19:25:14.404101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.404049    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:14.404206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.404109    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:25:14.433042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.433001    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.433042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.433046    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7l7\" (UniqueName: \"kubernetes.io/projected/a5c94ca5-e0a4-4556-864a-73283f8f4340-kube-api-access-sk7l7\") pod \"maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.433295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.433166    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.534479 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.534441    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.534479 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.534484    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7l7\" (UniqueName: \"kubernetes.io/projected/a5c94ca5-e0a4-4556-864a-73283f8f4340-kube-api-access-sk7l7\") pod \"maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.534700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.534552    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.535256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.535234    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.537201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.537163    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.556107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.555686    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7l7\" (UniqueName: \"kubernetes.io/projected/a5c94ca5-e0a4-4556-864a-73283f8f4340-kube-api-access-sk7l7\") pod \"maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:14.721515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:14.721434    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:15.024491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:15.024460    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"]
Oct 30 19:25:15.024596 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:25:15.024531    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c94ca5_e0a4_4556_864a_73283f8f4340.slice/crio-a60a19fe62bc36a52b230f88e01976fb7076dfa9ee2d2de86140173f681f1bbb WatchSource:0}: Error finding container a60a19fe62bc36a52b230f88e01976fb7076dfa9ee2d2de86140173f681f1bbb: Status 404 returned error can't find the container with id a60a19fe62bc36a52b230f88e01976fb7076dfa9ee2d2de86140173f681f1bbb
Oct 30 19:25:15.836140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:15.836096    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf" event={"ID":"a5c94ca5-e0a4-4556-864a-73283f8f4340","Type":"ContainerStarted","Data":"2e74b324a32afd276078429ea241619479816f5ef3d906075f40a7ea29b3c0e7"}
Oct 30 19:25:15.836535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:15.836164    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf" event={"ID":"a5c94ca5-e0a4-4556-864a-73283f8f4340","Type":"ContainerStarted","Data":"a60a19fe62bc36a52b230f88e01976fb7076dfa9ee2d2de86140173f681f1bbb"}
Oct 30 19:25:15.859428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:15.859372    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf" podStartSLOduration=1.859357197 podStartE2EDuration="1.859357197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:15.855224764 +0000 UTC m=+1097.852140146" watchObservedRunningTime="2026-10-30 19:25:15.859357197 +0000 UTC m=+1097.856272622"
Oct 30 19:25:16.884075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:16.882480    6777 generic.go:356] "Generic (PLEG): container finished" podID="a5c94ca5-e0a4-4556-864a-73283f8f4340" containerID="2e74b324a32afd276078429ea241619479816f5ef3d906075f40a7ea29b3c0e7" exitCode=0
Oct 30 19:25:16.884075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:16.882819    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf" event={"ID":"a5c94ca5-e0a4-4556-864a-73283f8f4340","Type":"ContainerDied","Data":"2e74b324a32afd276078429ea241619479816f5ef3d906075f40a7ea29b3c0e7"}
Oct 30 19:25:18.057861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.057827    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:18.210011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.209917    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-xks-ca\") pod \"a5c94ca5-e0a4-4556-864a-73283f8f4340\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") "
Oct 30 19:25:18.210210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.210112    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c94ca5-e0a4-4556-864a-73283f8f4340-kube-api-access-sk7l7\" (UniqueName: \"kubernetes.io/projected/a5c94ca5-e0a4-4556-864a-73283f8f4340-kube-api-access-sk7l7\") pod \"a5c94ca5-e0a4-4556-864a-73283f8f4340\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") "
Oct 30 19:25:18.210210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.210145    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-openshift-ca\") pod \"a5c94ca5-e0a4-4556-864a-73283f8f4340\" (UID: \"a5c94ca5-e0a4-4556-864a-73283f8f4340\") "
Oct 30 19:25:18.210546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.210515    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-openshift-ca" pod "a5c94ca5-e0a4-4556-864a-73283f8f4340" (UID: "a5c94ca5-e0a4-4556-864a-73283f8f4340"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:18.210858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.210838    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.212305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.212282    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-xks-ca" pod "a5c94ca5-e0a4-4556-864a-73283f8f4340" (UID: "a5c94ca5-e0a4-4556-864a-73283f8f4340"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:18.212421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.212403    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c94ca5-e0a4-4556-864a-73283f8f4340-kube-api-access-sk7l7" pod "a5c94ca5-e0a4-4556-864a-73283f8f4340" (UID: "a5c94ca5-e0a4-4556-864a-73283f8f4340"). InnerVolumeSpecName "kube-api-access-sk7l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:18.313302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.313258    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7l7\" (UniqueName: \"kubernetes.io/projected/a5c94ca5-e0a4-4556-864a-73283f8f4340-kube-api-access-sk7l7\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.313302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.313302    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c94ca5-e0a4-4556-864a-73283f8f4340-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.901298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.901261    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"
Oct 30 19:25:18.901298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.901286    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf" event={"ID":"a5c94ca5-e0a4-4556-864a-73283f8f4340","Type":"ContainerDied","Data":"a60a19fe62bc36a52b230f88e01976fb7076dfa9ee2d2de86140173f681f1bbb"}
Oct 30 19:25:18.901700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:18.901370    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a60a19fe62bc36a52b230f88e01976fb7076dfa9ee2d2de86140173f681f1bbb"
Oct 30 19:25:20.237712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.237671    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"]
Oct 30 19:25:20.238761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.238735    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a5c94ca5-e0a4-4556-864a-73283f8f4340" containerName="revoke-keys"
Oct 30 19:25:20.238885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.238779    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c94ca5-e0a4-4556-864a-73283f8f4340" containerName="revoke-keys"
Oct 30 19:25:20.239714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.239681    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c94ca5-e0a4-4556-864a-73283f8f4340" containerName="revoke-keys"
Oct 30 19:25:20.244981 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.244953    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.262692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.262504    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"]
Oct 30 19:25:20.376583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.376525    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.376905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.376877    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trb84\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-kube-api-access-trb84\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.377210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.377183    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.377328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.377287    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.377516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.377351    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.377598 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.377559    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.377669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.377645    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eac6694-e9ee-4636-8521-dca300646ba2-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.377736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.377676    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.377736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.377702    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eac6694-e9ee-4636-8521-dca300646ba2-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.391290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.391259    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"]
Oct 30 19:25:20.396068 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.396036    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.428867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.428824    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"]
Oct 30 19:25:20.479957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.479916    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.480185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480155    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.480185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480247    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.480472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480324    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.480472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480420    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.480630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480529    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eac6694-e9ee-4636-8521-dca300646ba2-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.480630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480568    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.480630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480596    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eac6694-e9ee-4636-8521-dca300646ba2-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.481029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480692    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.481029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480751    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trb84\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-kube-api-access-trb84\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.481029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480787    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.481029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.480824    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.481883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.481576    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.481982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.481961    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.482194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.482167    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eac6694-e9ee-4636-8521-dca300646ba2-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.482309 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.482291    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.482376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.482334    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gvc\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-kube-api-access-b4gvc\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.482431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.482396    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.482488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.482437    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.482488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.482472    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.482598 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.482495    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.482937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.482909    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.483275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.483251    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.484628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.484582    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.488480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.488420    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eac6694-e9ee-4636-8521-dca300646ba2-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.494946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.494915    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trb84\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-kube-api-access-trb84\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.495086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.495042    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.575487 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.575446    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:20.584807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.584321    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.584807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.584403    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.584807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.584441    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.584807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.584616    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.584807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.584706    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gvc\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-kube-api-access-b4gvc\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.584807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.584774    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.584807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.584804    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.585421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.584825    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.585421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.584875    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.585421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.585384    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.585648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.585629    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.585961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.585942    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.587537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.587514    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.588303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.588281    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.591097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.591033    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.591749 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.591724    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.601425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.601389    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.601729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.601702    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gvc\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-kube-api-access-b4gvc\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:20.750430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:20.750330    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:21.229823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:21.229766    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"]
Oct 30 19:25:21.262614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:21.262569    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:21.263272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:21.263242    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:21.267212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:21.267173    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:21.375983 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:21.375950    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"]
Oct 30 19:25:21.380286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:21.379433    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:21.380286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:21.379526    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:21.380286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:21.379576    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:22.399704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.399663    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" event={"ID":"44c4a8a2-fc32-4c4c-ae35-27d7afda27db","Type":"ContainerStarted","Data":"f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66"}
Oct 30 19:25:22.401471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.401431    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" event={"ID":"44c4a8a2-fc32-4c4c-ae35-27d7afda27db","Type":"ContainerStarted","Data":"6beabc99ccb08d50300a453e02cf3efa9570ef2020a27addeeb15b53ca6af626"}
Oct 30 19:25:22.405238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.405204    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" event={"ID":"2eac6694-e9ee-4636-8521-dca300646ba2","Type":"ContainerStarted","Data":"9deb101b0be2e1c6072648462dc016eb8be082ee27fb9a6cd618f8d29fb31b81"}
Oct 30 19:25:22.406711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.406683    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" event={"ID":"2eac6694-e9ee-4636-8521-dca300646ba2","Type":"ContainerStarted","Data":"d9023768d82676110b4f8265d7ea4e09c90cfc9062344f7bac5271ee836f7919"}
Oct 30 19:25:22.519086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.516011    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podStartSLOduration=2.515988759 podStartE2EDuration="2.515988759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:22.432597149 +0000 UTC m=+1104.429512533" watchObservedRunningTime="2026-10-30 19:25:22.515988759 +0000 UTC m=+1104.512904145"
Oct 30 19:25:22.545328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.543749    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podStartSLOduration=2.543725412 podStartE2EDuration="2.543725412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:22.487963026 +0000 UTC m=+1104.484878413" watchObservedRunningTime="2026-10-30 19:25:22.543725412 +0000 UTC m=+1104.540640798"
Oct 30 19:25:22.575783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.575706    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:22.578043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.578001    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:22.578315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.578271    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:22.755152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.754625    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:22.755152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.754757    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:22.755152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.754815    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:22.848280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.848220    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:25:22.902006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.901955    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:22.917097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.914202    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:22.937596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:22.937548    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:23.097085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.096217    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:23.101110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.100164    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:23.384172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.383263    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:23.578081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.577444    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:23.578081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.577606    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:23.578081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.577641    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:23.719279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.719159    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:25:23.762521 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.762482    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:23.762818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.762789    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:23.850219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.850151    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:25:23.851152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.851125    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:25:23.915214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.915161    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:23.916098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.916051    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:25:23.946083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.918640    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:25:23.961544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.929206    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.043791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.972742    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:25:24.045682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:23.973100    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.045682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.044819    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:25:24.139082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.114459    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.186710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.146076    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:25:24.367165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.339375    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.386475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.385465    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.604100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.582518    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:24.604100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.582600    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:24.606505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.606478    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:25:24.634723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.634198    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.736142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.727843    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.737280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.736672    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:25:24.762190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.762153    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:24.762393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.762214    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:24.875091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.851617    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.934983 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.934862    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:25:24.996082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:24.958142    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.047090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.047026    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.061642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.061598    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:25:25.074083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.064926    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:25:25.244144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.152164    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:25:25.456085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.455863    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:25:25.637437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.637391    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.706834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.706799    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:25:25.850260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.816198    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:25:25.903625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.903522    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:25.903921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.903886    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:25.926334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.926262    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:25.926667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:25.926617    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:26.098730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:26.098684    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.099802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:26.099766    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.301403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:26.301351    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.804148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:26.778850    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:26.924094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:26.888731    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.924094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:26.896480    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.924094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:26.898021    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.955508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:26.955451    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:25:27.092166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.090123    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:27.092166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.090205    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:27.092166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.091837    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:27.092166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.091897    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:27.100330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.093885    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:27.130563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.099113    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:27.243223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.243168    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:27.617860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.595664    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:27.796087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.788296    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 30 19:25:27.897151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.895346    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:27.901087 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:25:27.898494    6777 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.619s"
Oct 30 19:25:27.911591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.902487    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:27.911591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.902916    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:25:27.911591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:27.903889    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:25:28.028998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.003513    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:28.037194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.037159    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:28.062172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.062127    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:28.116326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.116278    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Oct 30 19:25:28.135271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.135233    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:28.139165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.139120    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:28.157139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.153482    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Oct 30 19:25:28.157391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.156652    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:25:28.175128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.161482    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:28.282084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.281095    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:28.321215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.313186    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:28.506154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.491638    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"]
Oct 30 19:25:28.615110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.607105    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:28.615110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.607175    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:28.647082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.646126    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:28.665120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.664911    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:25:28.834918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.817209    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:28.834918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:28.817369    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:29.019083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.015260    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:29.019083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.016287    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:29.019083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.016928    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:25:29.019083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.017162    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:25:29.052049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.033560    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Oct 30 19:25:29.052049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.035719    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:25:29.206147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.206035    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:25:29.284248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.284187    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:29.597869 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.597828    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:29.598509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.598474    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:29.663878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.663231    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:29.755559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.750808    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:29.755840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:29.755803    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:30.126218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.023168    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:25:30.126218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.023561    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:25:30.126218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.023809    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:25:30.126218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.024052    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:25:30.126218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.046614    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:30.127529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.127494    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:25:30.232028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.231978    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:30.337955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.337909    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:25:30.607088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.590843    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:30.637132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.637087    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:30.666691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.666626    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:30.840435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.840401    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"]
Oct 30 19:25:30.840860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.840837    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c74535f-dn5kf"]
Oct 30 19:25:30.889372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.857544    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:30.889372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:30.875825    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:31.019278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.019226    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:25:31.036229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.036175    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:25:31.036897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.036691    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:31.037589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.036813    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:25:31.047720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.047665    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:31.087952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.087898    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:31.146506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.146394    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:31.147587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.147562    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:25:31.154687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.148045    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:25:31.333416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.271437    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:25:31.340221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.340176    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:31.434090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.433948    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 30 19:25:31.620084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.595321    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:31.620084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.595400    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:31.686042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.685927    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:31.755250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.755213    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:31.755527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.755495    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:32.032145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:31.991102    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"]
Oct 30 19:25:32.048913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.048878    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp" podUID="f418d9df-8c20-45bc-913b-d63999cf42a1" containerName="istio-proxy" containerID="cri-o://510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850" gracePeriod=30
Oct 30 19:25:32.097082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.047841    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:32.112091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.048020    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:32.217698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.217652    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:25:32.302136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.290159    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:25:32.399351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.399310    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c94ca5-e0a4-4556-864a-73283f8f4340" path="/var/lib/kubelet/pods/a5c94ca5-e0a4-4556-864a-73283f8f4340/volumes"
Oct 30 19:25:32.400150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.400128    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:25:32.408204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.408176    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:25:32.526744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.525638    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:25:32.605694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.605599    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:32.605694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.605658    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:32.611167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.611140    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:25:32.618320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.618296    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:25:32.689090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.688701    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:25:32.757747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.757711    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:32.796520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:32.795635    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:25:33.153110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:33.143126    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"]
Oct 30 19:25:33.212081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:33.208649    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80302abc-qwjx9"]
Oct 30 19:25:33.492436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:33.492324    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4" podUID="64cdf9ed-f694-4a32-89a1-9d9d20009076" containerName="payload-pre-processing" containerID="cri-o://b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34" gracePeriod=25
Oct 30 19:25:33.576499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:33.576463    6777 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:25:33.576773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:33.576740    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 19:25:33.904080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:33.902406    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:33.907119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:33.906575    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:33.922839 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:33.922813    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:25:34.006105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.006079    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:25:34.122312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.122271    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec1b2e5-8104-403a-93b9-90bb0b3432f5" path="/var/lib/kubelet/pods/bec1b2e5-8104-403a-93b9-90bb0b3432f5/volumes"
Oct 30 19:25:34.143100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.136745    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64cdf9ed-f694-4a32-89a1-9d9d20009076-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cdf9ed-f694-4a32-89a1-9d9d20009076-plugins-config-volume\") pod \"64cdf9ed-f694-4a32-89a1-9d9d20009076\" (UID: \"64cdf9ed-f694-4a32-89a1-9d9d20009076\") "
Oct 30 19:25:34.143100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.137291    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cdf9ed-f694-4a32-89a1-9d9d20009076-kube-api-access-bdr5v\" (UniqueName: \"kubernetes.io/projected/64cdf9ed-f694-4a32-89a1-9d9d20009076-kube-api-access-bdr5v\") pod \"64cdf9ed-f694-4a32-89a1-9d9d20009076\" (UID: \"64cdf9ed-f694-4a32-89a1-9d9d20009076\") "
Oct 30 19:25:34.143100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.139016    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cdf9ed-f694-4a32-89a1-9d9d20009076-plugins-config-volume" pod "64cdf9ed-f694-4a32-89a1-9d9d20009076" (UID: "64cdf9ed-f694-4a32-89a1-9d9d20009076"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:34.212102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.212041    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cdf9ed-f694-4a32-89a1-9d9d20009076-kube-api-access-bdr5v" pod "64cdf9ed-f694-4a32-89a1-9d9d20009076" (UID: "64cdf9ed-f694-4a32-89a1-9d9d20009076"). InnerVolumeSpecName "kube-api-access-bdr5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:34.267083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.248096    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdr5v\" (UniqueName: \"kubernetes.io/projected/64cdf9ed-f694-4a32-89a1-9d9d20009076-kube-api-access-bdr5v\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:34.267373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.267344    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cdf9ed-f694-4a32-89a1-9d9d20009076-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:34.320379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.317819    6777 generic.go:356] "Generic (PLEG): container finished" podID="64cdf9ed-f694-4a32-89a1-9d9d20009076" containerID="b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34" exitCode=0
Oct 30 19:25:34.320379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.319275    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"
Oct 30 19:25:34.323381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.323346    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4" event={"ID":"64cdf9ed-f694-4a32-89a1-9d9d20009076","Type":"ContainerDied","Data":"b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34"}
Oct 30 19:25:34.324243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.324211    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4" event={"ID":"64cdf9ed-f694-4a32-89a1-9d9d20009076","Type":"ContainerDied","Data":"01f7eb13e0abbe1549c326f190acf715c5cf5b8efa6781ee93a44bd69ff30bc9"}
Oct 30 19:25:34.324890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.324421    6777 scope.go:122] "RemoveContainer" containerID="b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34"
Oct 30 19:25:34.390121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.387142    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"]
Oct 30 19:25:34.434188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.434083    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c74535f-7f9967fff9-mbjb4"]
Oct 30 19:25:34.483280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.471243    6777 scope.go:122] "RemoveContainer" containerID="b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34"
Oct 30 19:25:34.483280 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:25:34.471672    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34\": container with ID starting with b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34 not found: ID does not exist" containerID="b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34"
Oct 30 19:25:34.483280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.471709    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34"} err="failed to get container status \"b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34\": rpc error: code = NotFound desc = could not find container \"b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34\": container with ID starting with b7972eca08ff39ac671b68135b49c72a95fdfaf089365205f47d597762c72e34 not found: ID does not exist"
Oct 30 19:25:34.965529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.965445    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:34.965991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.965908    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:34.966930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:34.966906    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:25:36.114418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.114385    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64cdf9ed-f694-4a32-89a1-9d9d20009076" path="/var/lib/kubelet/pods/64cdf9ed-f694-4a32-89a1-9d9d20009076/volumes"
Oct 30 19:25:36.373032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.372946    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"]
Oct 30 19:25:36.373652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.373631    6777 cpu_manager.go:414] "containerMap: removing container" podUID="64cdf9ed-f694-4a32-89a1-9d9d20009076" containerName="payload-pre-processing"
Oct 30 19:25:36.373652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.373648    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cdf9ed-f694-4a32-89a1-9d9d20009076" containerName="payload-pre-processing"
Oct 30 19:25:36.373861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.373785    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cdf9ed-f694-4a32-89a1-9d9d20009076" containerName="payload-pre-processing"
Oct 30 19:25:36.379379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.379338    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.383807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.383780    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:36.384960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.384935    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:25:36.386278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.386244    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"]
Oct 30 19:25:36.434512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.434458    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrj8\" (UniqueName: \"kubernetes.io/projected/8b3b026c-5c09-45b7-8e7c-bcb350528439-kube-api-access-kkrj8\") pod \"maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.434841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.434526    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.434841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.434654    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.535975 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.535940    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.536238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.536220    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrj8\" (UniqueName: \"kubernetes.io/projected/8b3b026c-5c09-45b7-8e7c-bcb350528439-kube-api-access-kkrj8\") pod \"maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.536303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.536247    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.536895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.536821    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.539252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.539120    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.549817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.549781    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrj8\" (UniqueName: \"kubernetes.io/projected/8b3b026c-5c09-45b7-8e7c-bcb350528439-kube-api-access-kkrj8\") pod \"maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:36.723702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:36.723610    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:37.034775 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:25:37.034736    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3b026c_5c09_45b7_8e7c_bcb350528439.slice/crio-932aac080dcc9d44e73949f90ac87d2fc2efd985a31cdc0a93c914fd4ab125eb WatchSource:0}: Error finding container 932aac080dcc9d44e73949f90ac87d2fc2efd985a31cdc0a93c914fd4ab125eb: Status 404 returned error can't find the container with id 932aac080dcc9d44e73949f90ac87d2fc2efd985a31cdc0a93c914fd4ab125eb
Oct 30 19:25:37.037091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:37.037036    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"]
Oct 30 19:25:37.371122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:37.371092    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx" event={"ID":"8b3b026c-5c09-45b7-8e7c-bcb350528439","Type":"ContainerStarted","Data":"932aac080dcc9d44e73949f90ac87d2fc2efd985a31cdc0a93c914fd4ab125eb"}
Oct 30 19:25:38.383227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:38.383190    6777 generic.go:356] "Generic (PLEG): container finished" podID="8b3b026c-5c09-45b7-8e7c-bcb350528439" containerID="4b04675224ee394fad7ee8e03d5bfb0e52702ecce5ed80d366e5766b120f0e13" exitCode=0
Oct 30 19:25:38.383675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:38.383277    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx" event={"ID":"8b3b026c-5c09-45b7-8e7c-bcb350528439","Type":"ContainerDied","Data":"4b04675224ee394fad7ee8e03d5bfb0e52702ecce5ed80d366e5766b120f0e13"}
Oct 30 19:25:39.076621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.076591    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:25:39.201926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.201889    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-certs\") pod \"f418d9df-8c20-45bc-913b-d63999cf42a1\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") "
Oct 30 19:25:39.202134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.201973    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-data\") pod \"f418d9df-8c20-45bc-913b-d63999cf42a1\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") "
Oct 30 19:25:39.202134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202026    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-envoy\") pod \"f418d9df-8c20-45bc-913b-d63999cf42a1\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") "
Oct 30 19:25:39.202134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202129    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-token\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-token\") pod \"f418d9df-8c20-45bc-913b-d63999cf42a1\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") "
Oct 30 19:25:39.202249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202190    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-socket\") pod \"f418d9df-8c20-45bc-913b-d63999cf42a1\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") "
Oct 30 19:25:39.202249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202215    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-certs" pod "f418d9df-8c20-45bc-913b-d63999cf42a1" (UID: "f418d9df-8c20-45bc-913b-d63999cf42a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:39.202249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202232    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f418d9df-8c20-45bc-913b-d63999cf42a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f418d9df-8c20-45bc-913b-d63999cf42a1-istiod-ca-cert\") pod \"f418d9df-8c20-45bc-913b-d63999cf42a1\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") "
Oct 30 19:25:39.202407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202247    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-data" pod "f418d9df-8c20-45bc-913b-d63999cf42a1" (UID: "f418d9df-8c20-45bc-913b-d63999cf42a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:39.202407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202344    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-kube-api-access-zjl59\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-kube-api-access-zjl59\") pod \"f418d9df-8c20-45bc-913b-d63999cf42a1\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") "
Oct 30 19:25:39.202511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202433    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-podinfo\") pod \"f418d9df-8c20-45bc-913b-d63999cf42a1\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") "
Oct 30 19:25:39.202570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202501    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-socket" pod "f418d9df-8c20-45bc-913b-d63999cf42a1" (UID: "f418d9df-8c20-45bc-913b-d63999cf42a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:39.202570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202527    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-credential-socket\") pod \"f418d9df-8c20-45bc-913b-d63999cf42a1\" (UID: \"f418d9df-8c20-45bc-913b-d63999cf42a1\") "
Oct 30 19:25:39.202672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.202598    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f418d9df-8c20-45bc-913b-d63999cf42a1-istiod-ca-cert" pod "f418d9df-8c20-45bc-913b-d63999cf42a1" (UID: "f418d9df-8c20-45bc-913b-d63999cf42a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:39.203119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.203082    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-credential-socket" pod "f418d9df-8c20-45bc-913b-d63999cf42a1" (UID: "f418d9df-8c20-45bc-913b-d63999cf42a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:39.204669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.204640    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.204774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.204680    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.204774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.204698    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.204774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.204712    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.204774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.204731    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f418d9df-8c20-45bc-913b-d63999cf42a1-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.205291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.205210    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-envoy" pod "f418d9df-8c20-45bc-913b-d63999cf42a1" (UID: "f418d9df-8c20-45bc-913b-d63999cf42a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:39.217483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.217453    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-podinfo" pod "f418d9df-8c20-45bc-913b-d63999cf42a1" (UID: "f418d9df-8c20-45bc-913b-d63999cf42a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:25:39.217587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.217502    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-token" pod "f418d9df-8c20-45bc-913b-d63999cf42a1" (UID: "f418d9df-8c20-45bc-913b-d63999cf42a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:39.217672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.217656    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-kube-api-access-zjl59" pod "f418d9df-8c20-45bc-913b-d63999cf42a1" (UID: "f418d9df-8c20-45bc-913b-d63999cf42a1"). InnerVolumeSpecName "kube-api-access-zjl59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:39.306422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.306387    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.306422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.306415    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.306422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.306427    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjl59\" (UniqueName: \"kubernetes.io/projected/f418d9df-8c20-45bc-913b-d63999cf42a1-kube-api-access-zjl59\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.306656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.306437    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f418d9df-8c20-45bc-913b-d63999cf42a1-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.393307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.393271    6777 generic.go:356] "Generic (PLEG): container finished" podID="f418d9df-8c20-45bc-913b-d63999cf42a1" containerID="510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850" exitCode=0
Oct 30 19:25:39.393816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.393349    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"
Oct 30 19:25:39.393816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.393358    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp" event={"ID":"f418d9df-8c20-45bc-913b-d63999cf42a1","Type":"ContainerDied","Data":"510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850"}
Oct 30 19:25:39.393816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.393464    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp" event={"ID":"f418d9df-8c20-45bc-913b-d63999cf42a1","Type":"ContainerDied","Data":"8d8bf0477a2a3204f65ce2180b31482ccdb9d3722e5bdef41c8e8f3498730f8f"}
Oct 30 19:25:39.393816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.393498    6777 scope.go:122] "RemoveContainer" containerID="510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850"
Oct 30 19:25:39.407689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.407667    6777 scope.go:122] "RemoveContainer" containerID="510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850"
Oct 30 19:25:39.408039 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:25:39.408012    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850\": container with ID starting with 510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850 not found: ID does not exist" containerID="510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850"
Oct 30 19:25:39.408151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.408050    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850"} err="failed to get container status \"510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850\": rpc error: code = NotFound desc = could not find container \"510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850\": container with ID starting with 510f96d0be2b7a29ff0091a212c417ba973819b8970dfeed697e34f5246c7850 not found: ID does not exist"
Oct 30 19:25:39.422037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.422011    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"]
Oct 30 19:25:39.427473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.427446    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7c74535f-openshift-default-f55494c4f-fwpmp"]
Oct 30 19:25:39.543854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.543828    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:39.610557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.610453    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-openshift-ca\") pod \"8b3b026c-5c09-45b7-8e7c-bcb350528439\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") "
Oct 30 19:25:39.610557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.610505    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3b026c-5c09-45b7-8e7c-bcb350528439-kube-api-access-kkrj8\" (UniqueName: \"kubernetes.io/projected/8b3b026c-5c09-45b7-8e7c-bcb350528439-kube-api-access-kkrj8\") pod \"8b3b026c-5c09-45b7-8e7c-bcb350528439\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") "
Oct 30 19:25:39.610874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.610849    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-xks-ca\") pod \"8b3b026c-5c09-45b7-8e7c-bcb350528439\" (UID: \"8b3b026c-5c09-45b7-8e7c-bcb350528439\") "
Oct 30 19:25:39.610946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.610875    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-openshift-ca" pod "8b3b026c-5c09-45b7-8e7c-bcb350528439" (UID: "8b3b026c-5c09-45b7-8e7c-bcb350528439"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:39.611592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.611566    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.612954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.612926    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-xks-ca" pod "8b3b026c-5c09-45b7-8e7c-bcb350528439" (UID: "8b3b026c-5c09-45b7-8e7c-bcb350528439"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:39.613271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.613251    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3b026c-5c09-45b7-8e7c-bcb350528439-kube-api-access-kkrj8" pod "8b3b026c-5c09-45b7-8e7c-bcb350528439" (UID: "8b3b026c-5c09-45b7-8e7c-bcb350528439"). InnerVolumeSpecName "kube-api-access-kkrj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:39.713032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.712994    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3b026c-5c09-45b7-8e7c-bcb350528439-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.713032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:39.713028    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrj8\" (UniqueName: \"kubernetes.io/projected/8b3b026c-5c09-45b7-8e7c-bcb350528439-kube-api-access-kkrj8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:40.115902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:40.115866    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f418d9df-8c20-45bc-913b-d63999cf42a1" path="/var/lib/kubelet/pods/f418d9df-8c20-45bc-913b-d63999cf42a1/volumes"
Oct 30 19:25:40.403124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:40.403008    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"
Oct 30 19:25:40.403124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:40.403019    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx" event={"ID":"8b3b026c-5c09-45b7-8e7c-bcb350528439","Type":"ContainerDied","Data":"932aac080dcc9d44e73949f90ac87d2fc2efd985a31cdc0a93c914fd4ab125eb"}
Oct 30 19:25:40.403124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:40.403088    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="932aac080dcc9d44e73949f90ac87d2fc2efd985a31cdc0a93c914fd4ab125eb"
Oct 30 19:25:41.993572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:41.993531    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"]
Oct 30 19:25:41.994643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:41.994614    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8b3b026c-5c09-45b7-8e7c-bcb350528439" containerName="revoke-keys"
Oct 30 19:25:41.994643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:41.994642    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3b026c-5c09-45b7-8e7c-bcb350528439" containerName="revoke-keys"
Oct 30 19:25:41.994843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:41.994750    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f418d9df-8c20-45bc-913b-d63999cf42a1" containerName="istio-proxy"
Oct 30 19:25:41.994843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:41.994761    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f418d9df-8c20-45bc-913b-d63999cf42a1" containerName="istio-proxy"
Oct 30 19:25:41.995020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:41.995003    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3b026c-5c09-45b7-8e7c-bcb350528439" containerName="revoke-keys"
Oct 30 19:25:41.995093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:41.995031    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f418d9df-8c20-45bc-913b-d63999cf42a1" containerName="istio-proxy"
Oct 30 19:25:42.002295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.000621    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.011606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.011576    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-2405fb-openshift-default-dockercfg-mpx47\""
Oct 30 19:25:42.013849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.013822    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"]
Oct 30 19:25:42.178085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.163469    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.178085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.163626    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb479ae-22d3-45b4-99d4-d9377e881735-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.178085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.163742    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb479ae-22d3-45b4-99d4-d9377e881735-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.178085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.163771    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.178085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.163810    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.178085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.163857    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d9zm\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-kube-api-access-9d9zm\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.178085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.163906    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.178085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.163939    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.178085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.163985    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.271207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.269598    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb479ae-22d3-45b4-99d4-d9377e881735-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.271207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.269742    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb479ae-22d3-45b4-99d4-d9377e881735-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.271207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.269773    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.271207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.269812    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.271207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.269860    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d9zm\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-kube-api-access-9d9zm\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.271207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.269910    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.271207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.269945    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.271207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.269992    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.277041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.276270    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb479ae-22d3-45b4-99d4-d9377e881735-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.277041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.276559    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.281081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.279780    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.281081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.280706    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.281081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.280935    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.289084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.281566    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.301115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.300346    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.301115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.300562    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb479ae-22d3-45b4-99d4-d9377e881735-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.308237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.308199    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d9zm\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-kube-api-access-9d9zm\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.317890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.317280    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.324778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.324488    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:42.637904 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:25:42.637870    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb479ae_22d3_45b4_99d4_d9377e881735.slice/crio-09a556349f43660ce077dc970fbfa4e3482b06d95bf1548cc790795470ea0a33 WatchSource:0}: Error finding container 09a556349f43660ce077dc970fbfa4e3482b06d95bf1548cc790795470ea0a33: Status 404 returned error can't find the container with id 09a556349f43660ce077dc970fbfa4e3482b06d95bf1548cc790795470ea0a33
Oct 30 19:25:42.640811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.640773    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:42.640921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.640845    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:42.640921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.640878    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:42.642513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:42.642483    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"]
Oct 30 19:25:43.440677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:43.440635    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" event={"ID":"adb479ae-22d3-45b4-99d4-d9377e881735","Type":"ContainerStarted","Data":"ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3"}
Oct 30 19:25:43.441119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:43.440692    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" event={"ID":"adb479ae-22d3-45b4-99d4-d9377e881735","Type":"ContainerStarted","Data":"09a556349f43660ce077dc970fbfa4e3482b06d95bf1548cc790795470ea0a33"}
Oct 30 19:25:43.478349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:43.478270    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" podStartSLOduration=2.478246958 podStartE2EDuration="2.478246958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:43.466377864 +0000 UTC m=+1125.463293246" watchObservedRunningTime="2026-10-30 19:25:43.478246958 +0000 UTC m=+1125.475162344"
Oct 30 19:25:43.653339 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:25:43.653293    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:25:44.325805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:44.325770    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:44.332633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:44.332603    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:44.465199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:44.465164    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:44.466144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:44.466098    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:25:47.893081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:47.891197    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"]
Oct 30 19:25:47.901765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:47.900481    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c5ed0670-lbhhx"]
Oct 30 19:25:48.037729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:48.037631    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"]
Oct 30 19:25:48.053114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:48.046741    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12cbcb1-9zp5n"]
Oct 30 19:25:48.066734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:48.066698    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"]
Oct 30 19:25:48.067048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:48.067017    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb" podUID="eefcb615-55d2-426c-b13c-f83582f841d2" containerName="istio-proxy" containerID="cri-o://643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704" gracePeriod=30
Oct 30 19:25:48.126837 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:48.126794    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3b026c-5c09-45b7-8e7c-bcb350528439" path="/var/lib/kubelet/pods/8b3b026c-5c09-45b7-8e7c-bcb350528439/volumes"
Oct 30 19:25:48.127622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:48.127596    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c00d144-4aa7-43a2-9e99-21567b37d02e" path="/var/lib/kubelet/pods/8c00d144-4aa7-43a2-9e99-21567b37d02e/volumes"
Oct 30 19:25:48.819942 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:48.819905    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:25:49.048873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:49.048821    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"]
Oct 30 19:25:49.057465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:49.057418    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a12eb4-k4ndn"]
Oct 30 19:25:50.131377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:50.131344    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f7c9f5-bc4d-4397-8136-c595a93d6d7d" path="/var/lib/kubelet/pods/f9f7c9f5-bc4d-4397-8136-c595a93d6d7d/volumes"
Oct 30 19:25:53.350143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.350036    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:53.466533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466496    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-envoy\") pod \"eefcb615-55d2-426c-b13c-f83582f841d2\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") "
Oct 30 19:25:53.466727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466541    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-data\") pod \"eefcb615-55d2-426c-b13c-f83582f841d2\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") "
Oct 30 19:25:53.466727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466611    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-socket\") pod \"eefcb615-55d2-426c-b13c-f83582f841d2\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") "
Oct 30 19:25:53.466727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466703    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefcb615-55d2-426c-b13c-f83582f841d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefcb615-55d2-426c-b13c-f83582f841d2-istiod-ca-cert\") pod \"eefcb615-55d2-426c-b13c-f83582f841d2\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") "
Oct 30 19:25:53.466909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466737    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-certs\") pod \"eefcb615-55d2-426c-b13c-f83582f841d2\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") "
Oct 30 19:25:53.466909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466784    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-istio-token\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-istio-token\") pod \"eefcb615-55d2-426c-b13c-f83582f841d2\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") "
Oct 30 19:25:53.466909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466817    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-credential-socket\") pod \"eefcb615-55d2-426c-b13c-f83582f841d2\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") "
Oct 30 19:25:53.467117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466893    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-data" pod "eefcb615-55d2-426c-b13c-f83582f841d2" (UID: "eefcb615-55d2-426c-b13c-f83582f841d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:53.467117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466972    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-socket" pod "eefcb615-55d2-426c-b13c-f83582f841d2" (UID: "eefcb615-55d2-426c-b13c-f83582f841d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:53.467117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.466975    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-kube-api-access-zm44g\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-kube-api-access-zm44g\") pod \"eefcb615-55d2-426c-b13c-f83582f841d2\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") "
Oct 30 19:25:53.467282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.467122    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eefcb615-55d2-426c-b13c-f83582f841d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefcb615-55d2-426c-b13c-f83582f841d2-istio-podinfo\") pod \"eefcb615-55d2-426c-b13c-f83582f841d2\" (UID: \"eefcb615-55d2-426c-b13c-f83582f841d2\") "
Oct 30 19:25:53.467282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.467132    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-certs" pod "eefcb615-55d2-426c-b13c-f83582f841d2" (UID: "eefcb615-55d2-426c-b13c-f83582f841d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:53.467282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.467202    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-credential-socket" pod "eefcb615-55d2-426c-b13c-f83582f841d2" (UID: "eefcb615-55d2-426c-b13c-f83582f841d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:53.467282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.467218    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefcb615-55d2-426c-b13c-f83582f841d2-istiod-ca-cert" pod "eefcb615-55d2-426c-b13c-f83582f841d2" (UID: "eefcb615-55d2-426c-b13c-f83582f841d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:53.467914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.467890    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:53.467914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.467912    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:53.468115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.467921    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:53.468115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.467930    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefcb615-55d2-426c-b13c-f83582f841d2-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:53.468115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.467939    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:53.468999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.468974    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-envoy" pod "eefcb615-55d2-426c-b13c-f83582f841d2" (UID: "eefcb615-55d2-426c-b13c-f83582f841d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:53.469115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.469022    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-istio-token" pod "eefcb615-55d2-426c-b13c-f83582f841d2" (UID: "eefcb615-55d2-426c-b13c-f83582f841d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:53.469471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.469453    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eefcb615-55d2-426c-b13c-f83582f841d2-istio-podinfo" pod "eefcb615-55d2-426c-b13c-f83582f841d2" (UID: "eefcb615-55d2-426c-b13c-f83582f841d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:25:53.469518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.469490    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-kube-api-access-zm44g" pod "eefcb615-55d2-426c-b13c-f83582f841d2" (UID: "eefcb615-55d2-426c-b13c-f83582f841d2"). InnerVolumeSpecName "kube-api-access-zm44g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:53.546046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.546010    6777 generic.go:356] "Generic (PLEG): container finished" podID="eefcb615-55d2-426c-b13c-f83582f841d2" containerID="643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704" exitCode=0
Oct 30 19:25:53.546238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.546093    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"
Oct 30 19:25:53.546238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.546101    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb" event={"ID":"eefcb615-55d2-426c-b13c-f83582f841d2","Type":"ContainerDied","Data":"643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704"}
Oct 30 19:25:53.546238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.546183    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb" event={"ID":"eefcb615-55d2-426c-b13c-f83582f841d2","Type":"ContainerDied","Data":"5e94376b55e0aaf20e998bf3c8d231f225fa43e85d0636b2df49d6bc4b143dae"}
Oct 30 19:25:53.546238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.546205    6777 scope.go:122] "RemoveContainer" containerID="643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704"
Oct 30 19:25:53.558379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.558362    6777 scope.go:122] "RemoveContainer" containerID="643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704"
Oct 30 19:25:53.558705 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:25:53.558687    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704\": container with ID starting with 643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704 not found: ID does not exist" containerID="643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704"
Oct 30 19:25:53.558774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.558719    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704"} err="failed to get container status \"643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704\": rpc error: code = NotFound desc = could not find container \"643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704\": container with ID starting with 643fc33f195907aedcd982f40ff29c33cd440a3bfe124b8670898a779b39c704 not found: ID does not exist"
Oct 30 19:25:53.569223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.569195    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefcb615-55d2-426c-b13c-f83582f841d2-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:53.569223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.569217    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:53.569223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.569229    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm44g\" (UniqueName: \"kubernetes.io/projected/eefcb615-55d2-426c-b13c-f83582f841d2-kube-api-access-zm44g\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:53.569223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.569239    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefcb615-55d2-426c-b13c-f83582f841d2-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:53.572746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.572720    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"]
Oct 30 19:25:53.578451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:53.578430    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c5ed0670-openshift-default-5f977cdc49-rqrgb"]
Oct 30 19:25:54.115374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:54.115336    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefcb615-55d2-426c-b13c-f83582f841d2" path="/var/lib/kubelet/pods/eefcb615-55d2-426c-b13c-f83582f841d2/volumes"
Oct 30 19:25:58.965173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:58.965139    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"]
Oct 30 19:25:58.966126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:58.966104    6777 cpu_manager.go:414] "containerMap: removing container" podUID="eefcb615-55d2-426c-b13c-f83582f841d2" containerName="istio-proxy"
Oct 30 19:25:58.966126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:58.966125    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefcb615-55d2-426c-b13c-f83582f841d2" containerName="istio-proxy"
Oct 30 19:25:58.966392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:58.966378    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefcb615-55d2-426c-b13c-f83582f841d2" containerName="istio-proxy"
Oct 30 19:25:58.972853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:58.972831    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:58.985458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:58.983742    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"]
Oct 30 19:25:58.985458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:58.984084    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0d360546-openshift-default-dockercfg-kkqqz\""
Oct 30 19:25:59.090380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.090350    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280439f1-4695-45a6-b321-6f65f88cc8df-istiod-ca-cert\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.090554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.090426    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-certs\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.090554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.090454    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbnlm\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-kube-api-access-pbnlm\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.090634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.090579    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-credential-socket\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.090669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.090652    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-envoy\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.090700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.090669    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-data\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.090760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.090746    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-istio-token\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.090806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.090795    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-socket\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.090844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.090833    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280439f1-4695-45a6-b321-6f65f88cc8df-istio-podinfo\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.234152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.213129    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280439f1-4695-45a6-b321-6f65f88cc8df-istiod-ca-cert\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.234399    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280439f1-4695-45a6-b321-6f65f88cc8df-istiod-ca-cert\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.234558    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-certs\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.234591    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbnlm\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-kube-api-access-pbnlm\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.234761    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-credential-socket\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.234927    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-envoy\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.234957    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-data\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.235008    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-istio-token\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.235106    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-socket\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.235185    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280439f1-4695-45a6-b321-6f65f88cc8df-istio-podinfo\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.244820    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-data\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.258083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.251367    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-credential-socket\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.259036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.259006    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-certs\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.275116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.275083    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-socket\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.300983 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.300912    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280439f1-4695-45a6-b321-6f65f88cc8df-istio-podinfo\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.301450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.301401    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-envoy\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.314877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.314811    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbnlm\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-kube-api-access-pbnlm\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.324802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.315053    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-istio-token\") pod \"e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") " pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.597657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.597616    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:25:59.903739 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:25:59.903704    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280439f1_4695_45a6_b321_6f65f88cc8df.slice/crio-f01d1b9a2724236af488e70b6924ae317af103caf962b3cc6480e7660d89296f WatchSource:0}: Error finding container f01d1b9a2724236af488e70b6924ae317af103caf962b3cc6480e7660d89296f: Status 404 returned error can't find the container with id f01d1b9a2724236af488e70b6924ae317af103caf962b3cc6480e7660d89296f
Oct 30 19:25:59.904318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.904293    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"]
Oct 30 19:25:59.907070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.907025    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:59.907162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.907137    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:25:59.907212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:25:59.907170    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:00.030453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:00.030425    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"]
Oct 30 19:26:00.036962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:00.036930    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bee26fe-wwx94"]
Oct 30 19:26:00.129633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:00.129599    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cab960a-94b0-4964-b3b4-8a61c4a37a49" path="/var/lib/kubelet/pods/0cab960a-94b0-4964-b3b4-8a61c4a37a49/volumes"
Oct 30 19:26:00.611021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:00.610974    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" event={"ID":"280439f1-4695-45a6-b321-6f65f88cc8df","Type":"ContainerStarted","Data":"ee6e4e263af6b3c180b1b00c1c181b6612787f850251ec4f4aae9738a5eea846"}
Oct 30 19:26:00.611233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:00.611044    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" event={"ID":"280439f1-4695-45a6-b321-6f65f88cc8df","Type":"ContainerStarted","Data":"f01d1b9a2724236af488e70b6924ae317af103caf962b3cc6480e7660d89296f"}
Oct 30 19:26:00.639044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:00.638980    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" podStartSLOduration=2.638957651 podStartE2EDuration="2.638957651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:00.636447535 +0000 UTC m=+1142.633362921" watchObservedRunningTime="2026-10-30 19:26:00.638957651 +0000 UTC m=+1142.635873036"
Oct 30 19:26:01.598247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:01.598210    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:26:01.603783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:01.603759    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:26:01.619934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:01.619909    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:26:01.620753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:01.620736    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:26:02.039150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:02.039112    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"]
Oct 30 19:26:02.040365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:02.040343    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889ffa11-p4dmz"]
Oct 30 19:26:02.118461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:02.118420    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f2acce-36f7-49af-add9-5d36c81aafa0" path="/var/lib/kubelet/pods/98f2acce-36f7-49af-add9-5d36c81aafa0/volumes"
Oct 30 19:26:04.037574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:04.037519    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"]
Oct 30 19:26:04.042328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:04.042295    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec55f3b-nw6j5"]
Oct 30 19:26:04.114645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:04.114611    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead81709-4ac0-4d00-9166-92e267eb685c" path="/var/lib/kubelet/pods/ead81709-4ac0-4d00-9166-92e267eb685c/volumes"
Oct 30 19:26:12.057465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.057428    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"]
Oct 30 19:26:12.082708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.082665    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:12.088147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.088117    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"]
Oct 30 19:26:12.088893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.088854    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-1e7123\""
Oct 30 19:26:12.177275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.155248    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpfq\" (UniqueName: \"kubernetes.io/projected/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-kube-api-access-kdpfq\") pod \"payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz\" (UID: \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:12.177275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.176662    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz\" (UID: \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:12.307159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.278102    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz\" (UID: \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:12.315139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.307721    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpfq\" (UniqueName: \"kubernetes.io/projected/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-kube-api-access-kdpfq\") pod \"payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz\" (UID: \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:12.340153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.334576    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz\" (UID: \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:12.494152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.469031    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpfq\" (UniqueName: \"kubernetes.io/projected/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-kube-api-access-kdpfq\") pod \"payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz\" (UID: \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:12.705571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:12.705483    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:13.196311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:13.196277    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"]
Oct 30 19:26:13.643371 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:26:13.643325    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:26:13.838811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:13.838770    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz" event={"ID":"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4","Type":"ContainerStarted","Data":"6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57"}
Oct 30 19:26:13.839100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:13.838834    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz" event={"ID":"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4","Type":"ContainerStarted","Data":"6c74b985dfdf97bffef15200c1d131d34a8e38c639dbeb982cef14e2dcb74e64"}
Oct 30 19:26:13.839180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:13.839136    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:14.029712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:14.029650    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz" podStartSLOduration=2.029628898 podStartE2EDuration="2.029628898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:13.881315711 +0000 UTC m=+1155.878231097" watchObservedRunningTime="2026-10-30 19:26:14.029628898 +0000 UTC m=+1156.026544284"
Oct 30 19:26:14.033049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:14.033017    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"]
Oct 30 19:26:14.039906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:14.039876    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1788af-sqthl"]
Oct 30 19:26:14.115466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:14.115428    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34252ef3-3dd7-48c4-9bde-af9f27ffc3f7" path="/var/lib/kubelet/pods/34252ef3-3dd7-48c4-9bde-af9f27ffc3f7/volumes"
Oct 30 19:26:15.507240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.507198    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:26:15.507709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.507256    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:15.577557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.577513    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"]
Oct 30 19:26:15.598697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.598665    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"]
Oct 30 19:26:15.598921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.598790    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.613769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.613743    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e3586f-openshift-default-dockercfg-czzb7\""
Oct 30 19:26:15.678701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.678648    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.678890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.678767    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.678971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.678927    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.679039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.678993    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09659fd9-9629-41e5-9e07-e6680dfa05e4-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.679120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.679041    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.679120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.679085    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.679231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.679181    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.679231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.679221    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwgx\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-kube-api-access-vjwgx\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.679338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.679285    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.790083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.787460    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.790083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.787539    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwgx\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-kube-api-access-vjwgx\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.790083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.787620    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.790083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.787764    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.803590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.803549    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.808081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.804589    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.808081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.804756    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.808081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.804829    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09659fd9-9629-41e5-9e07-e6680dfa05e4-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.808081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.804972    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.808081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.805007    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.808081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.806269    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.817866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.814472    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.818820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.818386    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.828464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.820541    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.829463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.829414    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwgx\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-kube-api-access-vjwgx\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.841113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.829622    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09659fd9-9629-41e5-9e07-e6680dfa05e4-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.843602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.841571    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.895078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.868606    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:15.927105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:15.927024    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:16.266660 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:26:16.266625    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09659fd9_9629_41e5_9e07_e6680dfa05e4.slice/crio-3ba7cc4de3b682ad41cdf447d34c61b68661650ed41c8389ee1cc5540c9ecfc7 WatchSource:0}: Error finding container 3ba7cc4de3b682ad41cdf447d34c61b68661650ed41c8389ee1cc5540c9ecfc7: Status 404 returned error can't find the container with id 3ba7cc4de3b682ad41cdf447d34c61b68661650ed41c8389ee1cc5540c9ecfc7
Oct 30 19:26:16.268491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:16.268393    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"]
Oct 30 19:26:16.269825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:16.269783    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:16.269903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:16.269872    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:16.269946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:16.269918    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:16.896386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:16.896354    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" event={"ID":"09659fd9-9629-41e5-9e07-e6680dfa05e4","Type":"ContainerStarted","Data":"f1315dabf78f49aea8392827fd827a5e7722eaeb2bffbca7114380878fba13ef"}
Oct 30 19:26:16.896767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:16.896412    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" event={"ID":"09659fd9-9629-41e5-9e07-e6680dfa05e4","Type":"ContainerStarted","Data":"3ba7cc4de3b682ad41cdf447d34c61b68661650ed41c8389ee1cc5540c9ecfc7"}
Oct 30 19:26:16.926163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:16.926102    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" podStartSLOduration=1.926084685 podStartE2EDuration="1.926084685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:16.922510193 +0000 UTC m=+1158.919425577" watchObservedRunningTime="2026-10-30 19:26:16.926084685 +0000 UTC m=+1158.923000071"
Oct 30 19:26:16.927643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:16.927619    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:17.107309 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:17.107275    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"]
Oct 30 19:26:17.112728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:17.112589    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f62e31-fldfj"]
Oct 30 19:26:17.970956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:17.970908    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:18.125568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:18.125536    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9fd068c-11fc-4e31-b07b-c7653cbeb71c" path="/var/lib/kubelet/pods/c9fd068c-11fc-4e31-b07b-c7653cbeb71c/volumes"
Oct 30 19:26:18.916591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:18.916554    6777 generic.go:356] "Generic (PLEG): container finished" podID="adb479ae-22d3-45b4-99d4-d9377e881735" containerID="ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3" exitCode=0
Oct 30 19:26:18.916806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:18.916630    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" event={"ID":"adb479ae-22d3-45b4-99d4-d9377e881735","Type":"ContainerDied","Data":"ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3"}
Oct 30 19:26:18.917092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:18.917040    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:18.917596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:18.917573    6777 scope.go:122] "RemoveContainer" containerID="ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3"
Oct 30 19:26:18.917867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:18.917852    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:26:18.919239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:18.919212    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:18.919328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:18.919276    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:18.919328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:18.919308    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:19.929471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:19.929431    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" event={"ID":"adb479ae-22d3-45b4-99d4-d9377e881735","Type":"ContainerStarted","Data":"1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd"}
Oct 30 19:26:20.325545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.325516    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:26:20.325733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.325678    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:20.325784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.325727    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 19:26:20.599493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.599398    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"]
Oct 30 19:26:20.604018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.603997    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:20.629902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.629866    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518a909-687d-4d36-bdc7-58a016b1e661-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2\" (UID: \"1518a909-687d-4d36-bdc7-58a016b1e661\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:20.630217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.630193    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64b8q\" (UniqueName: \"kubernetes.io/projected/1518a909-687d-4d36-bdc7-58a016b1e661-kube-api-access-64b8q\") pod \"payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2\" (UID: \"1518a909-687d-4d36-bdc7-58a016b1e661\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:20.631897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.631864    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"]
Oct 30 19:26:20.651655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.646320    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e3586f-dockercfg-mkk5h\""
Oct 30 19:26:20.686449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.686415    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e3586f\""
Oct 30 19:26:20.941279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:20.851114    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518a909-687d-4d36-bdc7-58a016b1e661-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2\" (UID: \"1518a909-687d-4d36-bdc7-58a016b1e661\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:21.040079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:21.017722    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518a909-687d-4d36-bdc7-58a016b1e661-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2\" (UID: \"1518a909-687d-4d36-bdc7-58a016b1e661\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:21.040079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:21.017893    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64b8q\" (UniqueName: \"kubernetes.io/projected/1518a909-687d-4d36-bdc7-58a016b1e661-kube-api-access-64b8q\") pod \"payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2\" (UID: \"1518a909-687d-4d36-bdc7-58a016b1e661\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:21.041963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:21.041928    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64b8q\" (UniqueName: \"kubernetes.io/projected/1518a909-687d-4d36-bdc7-58a016b1e661-kube-api-access-64b8q\") pod \"payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2\" (UID: \"1518a909-687d-4d36-bdc7-58a016b1e661\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:21.230392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:21.230241    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:21.327706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:21.327136    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:21.327706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:21.327224    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 19:26:21.608626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:21.605248    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"]
Oct 30 19:26:22.031734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:22.031696    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2" event={"ID":"1518a909-687d-4d36-bdc7-58a016b1e661","Type":"ContainerStarted","Data":"f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397"}
Oct 30 19:26:22.032145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:22.031759    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2" event={"ID":"1518a909-687d-4d36-bdc7-58a016b1e661","Type":"ContainerStarted","Data":"3a5f26e50b545d7b4ab766e5c9a4f553ee288cf050c4827a2d1df427f22a307d"}
Oct 30 19:26:22.032145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:22.031783    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:22.078074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:22.078003    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2" podStartSLOduration=2.077987675 podStartE2EDuration="2.077987675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:22.076027669 +0000 UTC m=+1164.072943054" watchObservedRunningTime="2026-10-30 19:26:22.077987675 +0000 UTC m=+1164.074903059"
Oct 30 19:26:22.326005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:22.325918    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:22.326005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:22.325980    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 19:26:23.325576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:23.325537    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:23.326184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:23.326141    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 19:26:24.326307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:24.326272    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:24.326737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:24.326327    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 19:26:24.873543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:24.873499    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:26:25.325985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:25.325949    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:25.326294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:25.326264    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 19:26:26.036183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:26.036143    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"]
Oct 30 19:26:26.040138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:26.040114    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d755ed8-69jbt"]
Oct 30 19:26:26.113863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:26.113821    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d646b6e-7ffa-4594-8638-365abfc3ce2d" path="/var/lib/kubelet/pods/8d646b6e-7ffa-4594-8638-365abfc3ce2d/volumes"
Oct 30 19:26:26.344842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:26.344176    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:26:26.346165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:26.345941    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:26:26.348461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:26.348378    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:26:33.053141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:33.053070    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:26:34.829001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:34.828957    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:26:38.031158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:38.031117    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"]
Oct 30 19:26:38.039907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:38.039877    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e43b6-wctbg"]
Oct 30 19:26:38.114538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:38.114502    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb" path="/var/lib/kubelet/pods/c8cadbd8-4940-4aa9-8eb7-52b845ff4aeb/volumes"
Oct 30 19:26:39.089812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.089623    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"]
Oct 30 19:26:39.098220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.098191    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"]
Oct 30 19:26:39.098387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.098330    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.107335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.107309    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-48b1a6a7-openshift-default-dockercfg-gwncx\""
Oct 30 19:26:39.197980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.197937    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969296cc-bcee-4ad7-8944-977164eac29c-istiod-ca-cert\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.198231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.197986    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-istio-token\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.198231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.198216    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-certs\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.198357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.198283    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-envoy\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.198475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.198442    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncvw\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-kube-api-access-gncvw\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.198629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.198613    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-data\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.198682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.198663    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-credential-socket\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.198743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.198685    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969296cc-bcee-4ad7-8944-977164eac29c-istio-podinfo\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.198848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.198822    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-socket\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.310163    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-credential-socket\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.310220    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969296cc-bcee-4ad7-8944-977164eac29c-istio-podinfo\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.310289    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-socket\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.310327    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969296cc-bcee-4ad7-8944-977164eac29c-istiod-ca-cert\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.310358    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-istio-token\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.310503    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-certs\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.310566    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-envoy\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.310692    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncvw\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-kube-api-access-gncvw\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.310842    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-data\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.311344    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-data\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.312594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.311598    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-credential-socket\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.324764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.323754    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-socket\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.324764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.324426    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969296cc-bcee-4ad7-8944-977164eac29c-istiod-ca-cert\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.325638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.325612    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-certs\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.369192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.369092    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969296cc-bcee-4ad7-8944-977164eac29c-istio-podinfo\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.369709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.369663    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-envoy\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.393077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.388613    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-istio-token\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.440511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.439847    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncvw\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-kube-api-access-gncvw\") pod \"e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") " pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:39.714509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:39.714421    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:40.024427 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:26:40.024401    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969296cc_bcee_4ad7_8944_977164eac29c.slice/crio-05d026a14432b9aa95c18e1e321577d1cec3786918ccd8176d0ab952317771b1 WatchSource:0}: Error finding container 05d026a14432b9aa95c18e1e321577d1cec3786918ccd8176d0ab952317771b1: Status 404 returned error can't find the container with id 05d026a14432b9aa95c18e1e321577d1cec3786918ccd8176d0ab952317771b1
Oct 30 19:26:40.025191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:40.025165    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"]
Oct 30 19:26:40.027511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:40.027482    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:40.027593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:40.027544    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:40.027593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:40.027573    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:40.421517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:40.421480    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" event={"ID":"969296cc-bcee-4ad7-8944-977164eac29c","Type":"ContainerStarted","Data":"c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca"}
Oct 30 19:26:40.421928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:40.421533    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" event={"ID":"969296cc-bcee-4ad7-8944-977164eac29c","Type":"ContainerStarted","Data":"05d026a14432b9aa95c18e1e321577d1cec3786918ccd8176d0ab952317771b1"}
Oct 30 19:26:40.470083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:40.469134    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podStartSLOduration=1.469113042 podStartE2EDuration="1.469113042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:40.466602018 +0000 UTC m=+1182.463517416" watchObservedRunningTime="2026-10-30 19:26:40.469113042 +0000 UTC m=+1182.466028430"
Oct 30 19:26:40.715740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:40.715627    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:41.763618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:41.763489    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:42.438635 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:42.438605    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:42.439486 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:42.439465    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:26:43.641585 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:26:43.641540    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:26:47.038754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:47.035720    6777 patch_prober.go:29] interesting pod/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": read tcp 10.134.0.2:35396->10.134.0.109:15021: read: connection reset by peer" start-of-body=
Oct 30 19:26:47.038754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:47.035781    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": read tcp 10.134.0.2:35396->10.134.0.109:15021: read: connection reset by peer"
Oct 30 19:26:47.514871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:47.514783    6777 generic.go:356] "Generic (PLEG): container finished" podID="280439f1-4695-45a6-b321-6f65f88cc8df" containerID="ee6e4e263af6b3c180b1b00c1c181b6612787f850251ec4f4aae9738a5eea846" exitCode=0
Oct 30 19:26:47.514871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:47.514845    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" event={"ID":"280439f1-4695-45a6-b321-6f65f88cc8df","Type":"ContainerDied","Data":"ee6e4e263af6b3c180b1b00c1c181b6612787f850251ec4f4aae9738a5eea846"}
Oct 30 19:26:47.515387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:47.515371    6777 scope.go:122] "RemoveContainer" containerID="ee6e4e263af6b3c180b1b00c1c181b6612787f850251ec4f4aae9738a5eea846"
Oct 30 19:26:47.517159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:47.517125    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:47.517307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:47.517196    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:47.517307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:47.517226    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:48.525047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:48.525005    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" event={"ID":"280439f1-4695-45a6-b321-6f65f88cc8df","Type":"ContainerStarted","Data":"468cad87600f5831a09e192c862d1152e8701b5035c6194dd144b07cf3fb8d88"}
Oct 30 19:26:48.598124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:48.598090    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:26:48.598313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:48.598264    6777 patch_prober.go:29] interesting pod/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 19:26:48.598366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:48.598316    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 19:26:49.598941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:49.598908    6777 patch_prober.go:29] interesting pod/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 19:26:49.599365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:49.598974    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 19:26:50.603081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:50.599865    6777 patch_prober.go:29] interesting pod/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 19:26:50.603081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:50.599942    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 19:26:51.598340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:51.598302    6777 patch_prober.go:29] interesting pod/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 19:26:51.598537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:51.598364    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 19:26:52.598753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:52.598718    6777 patch_prober.go:29] interesting pod/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 19:26:52.599205 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:52.598768    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 19:26:53.604081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.603643    6777 patch_prober.go:29] interesting pod/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 19:26:53.604081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.603721    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 19:26:53.921666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.900434    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"]
Oct 30 19:26:53.930893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.930864    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"]
Oct 30 19:26:53.931152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.931138    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"]
Oct 30 19:26:53.935447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.935419    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:53.935856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.935837    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"]
Oct 30 19:26:53.937022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.936905    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:53.976901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.954137    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-48b1a6a7-serving-cert\""
Oct 30 19:26:53.976901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:53.954937    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9a718a-openshift-default-dockercfg-rvmx4\""
Oct 30 19:26:54.006410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.006358    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-istio-token\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.006822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.006797    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-maas-api-tls\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.007129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.007105    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nvtd\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-kube-api-access-9nvtd\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.007297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.007279    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-socket\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.007416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.007403    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-credential-socket\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.007572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.007556    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-openshift-service-ca\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.007699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.007684    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-certs\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.007907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.007879    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb33245b-7109-4488-8a23-9633a63c42dc-istio-podinfo\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.007998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.007944    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-data\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.008071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.007999    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-metrics-certs\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.008071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.008035    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb33245b-7109-4488-8a23-9633a63c42dc-istiod-ca-cert\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.008255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.008100    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-envoy\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.008255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.008164    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb5kk\" (UniqueName: \"kubernetes.io/projected/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-kube-api-access-kb5kk\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.109689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.109653    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb5kk\" (UniqueName: \"kubernetes.io/projected/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-kube-api-access-kb5kk\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.109894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.109758    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-istio-token\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.109894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.109878    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-maas-api-tls\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.110018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.109929    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nvtd\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-kube-api-access-9nvtd\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.109955    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-socket\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.109983    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-credential-socket\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.109999    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-openshift-service-ca\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.110315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110039    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-certs\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110087    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb33245b-7109-4488-8a23-9633a63c42dc-istio-podinfo\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110139    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-data\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110199    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-metrics-certs\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.110315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110238    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb33245b-7109-4488-8a23-9633a63c42dc-istiod-ca-cert\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110296    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-envoy\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110465    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-socket\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110523    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-certs\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110816    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-data\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.110843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.110832    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-openshift-service-ca\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.111190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.111156    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-credential-socket\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.111365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.111343    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb33245b-7109-4488-8a23-9633a63c42dc-istiod-ca-cert\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.113190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.113164    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-envoy\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.113279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.113204    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-metrics-certs\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.114732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.114709    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb33245b-7109-4488-8a23-9633a63c42dc-istio-podinfo\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.116395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.116372    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-maas-api-tls\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.128884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.128851    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-istio-token\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.129311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.129287    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb5kk\" (UniqueName: \"kubernetes.io/projected/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-kube-api-access-kb5kk\") pod \"maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.129380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.129339    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nvtd\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-kube-api-access-9nvtd\") pod \"e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") " pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.310029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.309998    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:54.343283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.343250    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:54.658131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.657877    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:26:54.678241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.678210    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:26:54.698256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.693373    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:26:54.817316 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:26:54.817286    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ee0a79_5166_462f_8dea_7b2cc83eaf4a.slice/crio-a6f679a1a39c198a4cc9b866d2e71d1e8d75faa48a143c7ac174c4d56488b40f WatchSource:0}: Error finding container a6f679a1a39c198a4cc9b866d2e71d1e8d75faa48a143c7ac174c4d56488b40f: Status 404 returned error can't find the container with id a6f679a1a39c198a4cc9b866d2e71d1e8d75faa48a143c7ac174c4d56488b40f
Oct 30 19:26:54.818743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.818717    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"]
Oct 30 19:26:54.819377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.819358    6777 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:26:54.911774 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:26:54.911746    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb33245b_7109_4488_8a23_9633a63c42dc.slice/crio-08f7831cc56ebd1ae72b29194966c44579a586961c6a24c44d0d2d119e323fb7 WatchSource:0}: Error finding container 08f7831cc56ebd1ae72b29194966c44579a586961c6a24c44d0d2d119e323fb7: Status 404 returned error can't find the container with id 08f7831cc56ebd1ae72b29194966c44579a586961c6a24c44d0d2d119e323fb7
Oct 30 19:26:54.911934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.911907    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"]
Oct 30 19:26:54.915015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.914976    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:54.915142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.915054    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:54.915185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:54.915144    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:26:55.697293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:55.697197    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" event={"ID":"cb33245b-7109-4488-8a23-9633a63c42dc","Type":"ContainerStarted","Data":"00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e"}
Oct 30 19:26:55.697874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:55.697843    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" event={"ID":"cb33245b-7109-4488-8a23-9633a63c42dc","Type":"ContainerStarted","Data":"08f7831cc56ebd1ae72b29194966c44579a586961c6a24c44d0d2d119e323fb7"}
Oct 30 19:26:55.699588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:55.699564    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb" event={"ID":"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a","Type":"ContainerStarted","Data":"f6d2213ffa33f88c11fdf70b7359fc5e0c590481b6eac43070b961599820ee53"}
Oct 30 19:26:55.699697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:55.699614    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb" event={"ID":"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a","Type":"ContainerStarted","Data":"a6f679a1a39c198a4cc9b866d2e71d1e8d75faa48a143c7ac174c4d56488b40f"}
Oct 30 19:26:55.700040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:55.700018    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:26:55.732913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:55.732862    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" podStartSLOduration=2.7328477810000003 podStartE2EDuration="2.732847781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:55.729813388 +0000 UTC m=+1197.726728773" watchObservedRunningTime="2026-10-30 19:26:55.732847781 +0000 UTC m=+1197.729763166"
Oct 30 19:26:55.755158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:55.755108    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb" podStartSLOduration=2.243766992 podStartE2EDuration="2.755095491s" totalImagesPullingTime="511.328499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:55.752828742 +0000 UTC m=+1197.749744126" watchObservedRunningTime="2026-10-30 19:26:55.755095491 +0000 UTC m=+1197.752010915"
Oct 30 19:26:56.343649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:56.343608    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:56.350416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:56.350384    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:56.769265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:56.769175    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:56.771043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:56.771020    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:26:58.552081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:58.550181    6777 scope.go:122] "RemoveContainer" containerID="7e008d6f66251eb2156095ad618959b207c74e17859394e5d3e257c71956f550"
Oct 30 19:26:58.693368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:58.693331    6777 scope.go:122] "RemoveContainer" containerID="7302c8d4147e3f64d49075aa49fb4af76fafc749c256efec7f532f6dccf8d2af"
Oct 30 19:26:58.694979 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:26:58.694942    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e008d6f66251eb2156095ad618959b207c74e17859394e5d3e257c71956f550\": container with ID starting with 7e008d6f66251eb2156095ad618959b207c74e17859394e5d3e257c71956f550 not found: ID does not exist" containerID="7e008d6f66251eb2156095ad618959b207c74e17859394e5d3e257c71956f550"
Oct 30 19:26:58.695188 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:26:58.695165    6777 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"7e008d6f66251eb2156095ad618959b207c74e17859394e5d3e257c71956f550\": container with ID starting with 7e008d6f66251eb2156095ad618959b207c74e17859394e5d3e257c71956f550 not found: ID does not exist"
Oct 30 19:26:58.745544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:58.744583    6777 scope.go:122] "RemoveContainer" containerID="5ce88a239c1ed29d92f5b12be6930fa3deeee67ffba5181c5351dead829a93c8"
Oct 30 19:26:58.759844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:58.759823    6777 scope.go:122] "RemoveContainer" containerID="64e40ca3173f5205af2908f5c52f3be1a62fd2cc4f2c4f28ebedc71c1a66eb16"
Oct 30 19:26:59.851835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:59.851801    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"]
Oct 30 19:26:59.857987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:59.857968    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:26:59.865587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:59.865568    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9a718a-dockercfg-mdw8c\""
Oct 30 19:26:59.867199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:59.867178    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9a718a\""
Oct 30 19:26:59.958153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:26:59.945516    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"]
Oct 30 19:27:00.143462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:00.131266    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ef741-11df-4980-9405-c51808b647bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45\" (UID: \"ef1ef741-11df-4980-9405-c51808b647bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:27:00.143462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:00.131343    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptb2j\" (UniqueName: \"kubernetes.io/projected/ef1ef741-11df-4980-9405-c51808b647bc-kube-api-access-ptb2j\") pod \"payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45\" (UID: \"ef1ef741-11df-4980-9405-c51808b647bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:27:00.233393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:00.233340    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ef741-11df-4980-9405-c51808b647bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45\" (UID: \"ef1ef741-11df-4980-9405-c51808b647bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:27:00.233676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:00.233654    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptb2j\" (UniqueName: \"kubernetes.io/projected/ef1ef741-11df-4980-9405-c51808b647bc-kube-api-access-ptb2j\") pod \"payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45\" (UID: \"ef1ef741-11df-4980-9405-c51808b647bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:27:00.235214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:00.235192    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ef741-11df-4980-9405-c51808b647bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45\" (UID: \"ef1ef741-11df-4980-9405-c51808b647bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:27:00.249597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:00.249572    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptb2j\" (UniqueName: \"kubernetes.io/projected/ef1ef741-11df-4980-9405-c51808b647bc-kube-api-access-ptb2j\") pod \"payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45\" (UID: \"ef1ef741-11df-4980-9405-c51808b647bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:27:00.473603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:00.473522    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:27:00.758088 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:27:00.758024    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1ef741_11df_4980_9405_c51808b647bc.slice/crio-68aac96cc0ba959d0a5f366c08da8eee944be79fa1c30140e1a06eaf87769502 WatchSource:0}: Error finding container 68aac96cc0ba959d0a5f366c08da8eee944be79fa1c30140e1a06eaf87769502: Status 404 returned error can't find the container with id 68aac96cc0ba959d0a5f366c08da8eee944be79fa1c30140e1a06eaf87769502
Oct 30 19:27:00.761873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:00.761844    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"]
Oct 30 19:27:00.805875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:00.805843    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45" event={"ID":"ef1ef741-11df-4980-9405-c51808b647bc","Type":"ContainerStarted","Data":"68aac96cc0ba959d0a5f366c08da8eee944be79fa1c30140e1a06eaf87769502"}
Oct 30 19:27:01.774244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:01.774219    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:27:01.819728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:01.819690    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45" event={"ID":"ef1ef741-11df-4980-9405-c51808b647bc","Type":"ContainerStarted","Data":"604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f"}
Oct 30 19:27:01.819887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:01.819789    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:27:01.849229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:01.849165    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45" podStartSLOduration=2.849145688 podStartE2EDuration="2.849145688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:01.844083155 +0000 UTC m=+1203.840998545" watchObservedRunningTime="2026-10-30 19:27:01.849145688 +0000 UTC m=+1203.846061075"
Oct 30 19:27:02.548924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:02.548890    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:27:04.096405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:04.096372    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"]
Oct 30 19:27:04.097052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:04.097025    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c0b735e-2xsg5"]
Oct 30 19:27:04.135904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:04.134706    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beae642a-7921-4f8d-a903-7336dba1dc2a" path="/var/lib/kubelet/pods/beae642a-7921-4f8d-a903-7336dba1dc2a/volumes"
Oct 30 19:27:06.868083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:06.867343    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:06.923084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:06.915453    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:06.928946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:06.928201    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.094083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:07.093151    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.303083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:07.301154    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.506228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:07.506177    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.826082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:07.814141    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.906082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:07.905246    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.921402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:07.921156    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.928685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:07.928651    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.929934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:07.929914    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:27:07.938770 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:07.937965    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:08.022197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:08.017276    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded"
Oct 30 19:27:08.022197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:08.017510    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:27:08.118157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:08.103159    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:08.215426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:08.211290    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:27:08.239922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:08.239818    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:08.411138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:08.369264    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:08.518463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:08.518433    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:27:08.518707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:08.518521    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:27:12.831683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:12.831651    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:27:13.640777 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:27:13.640736    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:27:15.269489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:15.269399    6777 generic.go:356] "Generic (PLEG): container finished" podID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerID="f1315dabf78f49aea8392827fd827a5e7722eaeb2bffbca7114380878fba13ef" exitCode=0
Oct 30 19:27:15.269489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:15.269468    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" event={"ID":"09659fd9-9629-41e5-9e07-e6680dfa05e4","Type":"ContainerDied","Data":"f1315dabf78f49aea8392827fd827a5e7722eaeb2bffbca7114380878fba13ef"}
Oct 30 19:27:15.270113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:15.270049    6777 scope.go:122] "RemoveContainer" containerID="f1315dabf78f49aea8392827fd827a5e7722eaeb2bffbca7114380878fba13ef"
Oct 30 19:27:15.272425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:15.272389    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:15.272520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:15.272487    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:15.272566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:15.272537    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:16.312416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:16.312374    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" event={"ID":"09659fd9-9629-41e5-9e07-e6680dfa05e4","Type":"ContainerStarted","Data":"7e08bc99cef45be329c48e90944cf9ff4f72942ab3942515a46b63daee12700b"}
Oct 30 19:27:16.934566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:16.928220    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:27:16.934566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:16.931241    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:27:16.934566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:16.931310    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:27:16.988119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:16.988085    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"]
Oct 30 19:27:16.997973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:16.997944    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.012037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.012005    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b281d5-serving-cert\""
Oct 30 19:27:17.013347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.013322    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"]
Oct 30 19:27:17.025632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.025603    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.025814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.025794    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.025892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.025858    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a234178b-2e51-465e-9467-6f210c30799f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.025953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.025918    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75mmt\" (UniqueName: \"kubernetes.io/projected/a234178b-2e51-465e-9467-6f210c30799f-kube-api-access-75mmt\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.147102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.128386    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a234178b-2e51-465e-9467-6f210c30799f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.147517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.147486    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75mmt\" (UniqueName: \"kubernetes.io/projected/a234178b-2e51-465e-9467-6f210c30799f-kube-api-access-75mmt\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.152589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.148284    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.169100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.153019    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.169607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.137683    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a234178b-2e51-465e-9467-6f210c30799f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.197129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.194577    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75mmt\" (UniqueName: \"kubernetes.io/projected/a234178b-2e51-465e-9467-6f210c30799f-kube-api-access-75mmt\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.197129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.195153    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.197970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.197948    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.316964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.316932    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:17.647310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.647284    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"]
Oct 30 19:27:17.647649 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:27:17.647626    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda234178b_2e51_465e_9467_6f210c30799f.slice/crio-55e5b9bb29c71ea3d7e1d06aded0e379be526f3eb50a14cd2e68046139797b32 WatchSource:0}: Error finding container 55e5b9bb29c71ea3d7e1d06aded0e379be526f3eb50a14cd2e68046139797b32: Status 404 returned error can't find the container with id 55e5b9bb29c71ea3d7e1d06aded0e379be526f3eb50a14cd2e68046139797b32
Oct 30 19:27:17.936141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.935588    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:27:17.936141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:17.935650    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:27:18.333130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:18.333090    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x" event={"ID":"a234178b-2e51-465e-9467-6f210c30799f","Type":"ContainerStarted","Data":"81af924074e7e171bde928f49afc7f1d6fec306305ed80df98772c9f0d794be8"}
Oct 30 19:27:18.333501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:18.333164    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x" event={"ID":"a234178b-2e51-465e-9467-6f210c30799f","Type":"ContainerStarted","Data":"55e5b9bb29c71ea3d7e1d06aded0e379be526f3eb50a14cd2e68046139797b32"}
Oct 30 19:27:18.333501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:18.333194    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:18.375476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:18.375413    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x" podStartSLOduration=1.941106296 podStartE2EDuration="2.375394919s" totalImagesPullingTime="434.288623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:18.358529575 +0000 UTC m=+1220.355444959" watchObservedRunningTime="2026-10-30 19:27:18.375394919 +0000 UTC m=+1220.372310304"
Oct 30 19:27:18.917483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:18.917448    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:27:18.927926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:18.927895    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:27:18.928051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:18.927948    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:27:19.928160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:19.928120    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:27:19.928549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:19.928177    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:27:20.928349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:20.928315    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 19:27:20.928739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:20.928368    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 19:27:21.933162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:21.933131    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:27:21.934019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:21.933994    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:27:24.346860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:24.346829    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:27:31.133440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.133377    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"]
Oct 30 19:27:31.145248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.145212    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.154785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.154375    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9b2e27-openshift-default-dockercfg-9mmkm\""
Oct 30 19:27:31.165167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.165136    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"]
Oct 30 19:27:31.252677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.252634    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-podinfo\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.252932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.252911    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-certs\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.253153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.253131    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cb5dd9-9980-4623-9d2d-67f167639c95-istiod-ca-cert\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.253372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.253343    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-token\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.253589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.253570    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-credential-socket\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.253705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.253692    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9st5\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-kube-api-access-j9st5\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.253803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.253789    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-envoy\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.254009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.253990    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-socket\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.254157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.254138    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-data\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.373286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.372304    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-socket\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.373286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.372353    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-data\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.373286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.372599    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-podinfo\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.373286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.372654    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-certs\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.373286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.372719    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cb5dd9-9980-4623-9d2d-67f167639c95-istiod-ca-cert\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.373286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.372771    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-token\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.373286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.372843    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-credential-socket\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.373286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.372874    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9st5\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-kube-api-access-j9st5\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.373286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.372902    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-envoy\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.387488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.387413    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-socket\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.388001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.387941    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-data\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.393481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.388849    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-credential-socket\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.393481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.389104    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-certs\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.393481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.389715    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cb5dd9-9980-4623-9d2d-67f167639c95-istiod-ca-cert\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.407046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.405812    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-podinfo\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.412402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.410917    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-token\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.414739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.413276    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9st5\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-kube-api-access-j9st5\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.415447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.415417    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-envoy\") pod \"e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") " pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.470078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.469976    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:31.779290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.779260    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"]
Oct 30 19:27:31.779290 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:27:31.779281    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cb5dd9_9980_4623_9d2d_67f167639c95.slice/crio-03974c5700b6e1b045acd2f6c190d6a7001461cdc89bca8c5789ec9f273a911d WatchSource:0}: Error finding container 03974c5700b6e1b045acd2f6c190d6a7001461cdc89bca8c5789ec9f273a911d: Status 404 returned error can't find the container with id 03974c5700b6e1b045acd2f6c190d6a7001461cdc89bca8c5789ec9f273a911d
Oct 30 19:27:31.782779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.782743    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:31.782862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.782815    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:31.782862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:31.782851    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:32.574398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:32.566463    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" event={"ID":"81cb5dd9-9980-4623-9d2d-67f167639c95","Type":"ContainerStarted","Data":"91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec"}
Oct 30 19:27:32.574398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:32.566533    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" event={"ID":"81cb5dd9-9980-4623-9d2d-67f167639c95","Type":"ContainerStarted","Data":"03974c5700b6e1b045acd2f6c190d6a7001461cdc89bca8c5789ec9f273a911d"}
Oct 30 19:27:32.658575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:32.658512    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" podStartSLOduration=1.658494035 podStartE2EDuration="1.658494035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:32.610532632 +0000 UTC m=+1234.607448018" watchObservedRunningTime="2026-10-30 19:27:32.658494035 +0000 UTC m=+1234.655409421"
Oct 30 19:27:33.046389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:33.046352    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"]
Oct 30 19:27:33.057814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:33.057781    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c654b3-7nzlq"]
Oct 30 19:27:33.470682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:33.470593    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:33.477275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:33.477251    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:33.575358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:33.575329    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:33.576503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:33.576476    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:27:34.168080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:34.162247    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adba99d3-c9f9-43b4-89ef-1a27db198915" path="/var/lib/kubelet/pods/adba99d3-c9f9-43b4-89ef-1a27db198915/volumes"
Oct 30 19:27:39.312652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.312609    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"]
Oct 30 19:27:39.321887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.321856    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:39.343461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.343433    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8f3ff4\""
Oct 30 19:27:39.345865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.345831    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"]
Oct 30 19:27:39.384834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.376130    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8f3ff4-dockercfg-r7lkz\""
Oct 30 19:27:39.389124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.385702    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"]
Oct 30 19:27:39.394473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.394450    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:39.397815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.397791    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"]
Oct 30 19:27:39.401933 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.401905    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp\" (UID: \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:39.402183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.402163    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq5dv\" (UniqueName: \"kubernetes.io/projected/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-kube-api-access-nq5dv\") pod \"payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp\" (UID: \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:39.402362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.402346    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnk4f\" (UniqueName: \"kubernetes.io/projected/b9dcef83-3bad-4f42-81a8-3051ec762234-kube-api-access-fnk4f\") pod \"payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx\" (UID: \"b9dcef83-3bad-4f42-81a8-3051ec762234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:39.402605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.402590    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dcef83-3bad-4f42-81a8-3051ec762234-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx\" (UID: \"b9dcef83-3bad-4f42-81a8-3051ec762234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:39.684028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.679922    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp\" (UID: \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:39.684028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.680049    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq5dv\" (UniqueName: \"kubernetes.io/projected/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-kube-api-access-nq5dv\") pod \"payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp\" (UID: \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:39.684028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.680178    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnk4f\" (UniqueName: \"kubernetes.io/projected/b9dcef83-3bad-4f42-81a8-3051ec762234-kube-api-access-fnk4f\") pod \"payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx\" (UID: \"b9dcef83-3bad-4f42-81a8-3051ec762234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:39.684028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.680439    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dcef83-3bad-4f42-81a8-3051ec762234-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx\" (UID: \"b9dcef83-3bad-4f42-81a8-3051ec762234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:39.684028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.682300    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dcef83-3bad-4f42-81a8-3051ec762234-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx\" (UID: \"b9dcef83-3bad-4f42-81a8-3051ec762234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:39.728079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.725438    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp\" (UID: \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:39.784522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.784485    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq5dv\" (UniqueName: \"kubernetes.io/projected/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-kube-api-access-nq5dv\") pod \"payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp\" (UID: \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:39.791484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.791453    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:39.793368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.793332    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnk4f\" (UniqueName: \"kubernetes.io/projected/b9dcef83-3bad-4f42-81a8-3051ec762234-kube-api-access-fnk4f\") pod \"payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx\" (UID: \"b9dcef83-3bad-4f42-81a8-3051ec762234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:39.989992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:39.989273    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:40.181919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.181867    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"]
Oct 30 19:27:40.182815 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:27:40.182775    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93c51db_f93e_4cf5_aaa9_1a74a9b55fe2.slice/crio-dd9157752a133578e70687d21283741378a738357a596b00b8fef6a5123feb08 WatchSource:0}: Error finding container dd9157752a133578e70687d21283741378a738357a596b00b8fef6a5123feb08: Status 404 returned error can't find the container with id dd9157752a133578e70687d21283741378a738357a596b00b8fef6a5123feb08
Oct 30 19:27:40.408822 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:27:40.408792    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9dcef83_3bad_4f42_81a8_3051ec762234.slice/crio-5fc218c33a1e11ec14c77ddb1e6f506cd0cc2d7d7a987f89d9634670d3811221 WatchSource:0}: Error finding container 5fc218c33a1e11ec14c77ddb1e6f506cd0cc2d7d7a987f89d9634670d3811221: Status 404 returned error can't find the container with id 5fc218c33a1e11ec14c77ddb1e6f506cd0cc2d7d7a987f89d9634670d3811221
Oct 30 19:27:40.409514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.409456    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"]
Oct 30 19:27:40.826452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.826412    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx" event={"ID":"b9dcef83-3bad-4f42-81a8-3051ec762234","Type":"ContainerStarted","Data":"c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81"}
Oct 30 19:27:40.826649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.826473    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx" event={"ID":"b9dcef83-3bad-4f42-81a8-3051ec762234","Type":"ContainerStarted","Data":"5fc218c33a1e11ec14c77ddb1e6f506cd0cc2d7d7a987f89d9634670d3811221"}
Oct 30 19:27:40.826649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.826502    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:40.828438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.828415    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp" event={"ID":"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2","Type":"ContainerStarted","Data":"16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f"}
Oct 30 19:27:40.828552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.828456    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp" event={"ID":"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2","Type":"ContainerStarted","Data":"dd9157752a133578e70687d21283741378a738357a596b00b8fef6a5123feb08"}
Oct 30 19:27:40.828552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.828539    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:40.903259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.903207    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp" podStartSLOduration=1.903192431 podStartE2EDuration="1.903192431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:40.899162026 +0000 UTC m=+1242.896077411" watchObservedRunningTime="2026-10-30 19:27:40.903192431 +0000 UTC m=+1242.900107833"
Oct 30 19:27:40.903765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:40.903743    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx" podStartSLOduration=1.903736362 podStartE2EDuration="1.903736362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:40.877099574 +0000 UTC m=+1242.874014953" watchObservedRunningTime="2026-10-30 19:27:40.903736362 +0000 UTC m=+1242.900651804"
Oct 30 19:27:41.675986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.675893    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"]
Oct 30 19:27:41.682365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.682314    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.692940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.692319    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b281d5-openshift-default-dockercfg-tjwsd\""
Oct 30 19:27:41.698748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.698722    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"]
Oct 30 19:27:41.737899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.737843    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-certs\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.738146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.737912    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-istio-token\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.738146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.737954    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-credential-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.738146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.737989    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-envoy\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.738146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.738021    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46xz\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-kube-api-access-f46xz\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.738146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.738115    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.738516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.738396    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-data\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.738516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.738432    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdde26a-de3b-4a41-812a-cefa04196839-istio-podinfo\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.738516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.738485    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdde26a-de3b-4a41-812a-cefa04196839-istiod-ca-cert\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.840372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.840335    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-certs\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.840589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.840562    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-istio-token\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.840674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.840624    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-credential-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.840674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.840665    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-envoy\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.840787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.840691    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46xz\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-kube-api-access-f46xz\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.840787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.840752    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.840912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.840778    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-certs\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.840912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.840885    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-credential-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.841160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.841137    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-data\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.841266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.841174    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdde26a-de3b-4a41-812a-cefa04196839-istio-podinfo\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.841266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.841196    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.841266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.841208    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdde26a-de3b-4a41-812a-cefa04196839-istiod-ca-cert\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.841499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.841442    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-data\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.842034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.842005    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdde26a-de3b-4a41-812a-cefa04196839-istiod-ca-cert\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.843230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.843209    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-envoy\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.843710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.843689    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdde26a-de3b-4a41-812a-cefa04196839-istio-podinfo\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.853804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.853777    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46xz\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-kube-api-access-f46xz\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:41.884829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:41.884349    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-istio-token\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:42.022662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.022576    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:42.030126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.030047    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"]
Oct 30 19:27:42.034670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.034639    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bcee05-fjdfj"]
Oct 30 19:27:42.118395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.118362    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c24133-5d0f-43ac-a702-6fd1fc69f5ae" path="/var/lib/kubelet/pods/50c24133-5d0f-43ac-a702-6fd1fc69f5ae/volumes"
Oct 30 19:27:42.353445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.353073    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"]
Oct 30 19:27:42.353445 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:27:42.353272    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdde26a_de3b_4a41_812a_cefa04196839.slice/crio-108c7a46e4c16b799827a7063df5cdcc5a20a8665c7702897421fb1345bcd70b WatchSource:0}: Error finding container 108c7a46e4c16b799827a7063df5cdcc5a20a8665c7702897421fb1345bcd70b: Status 404 returned error can't find the container with id 108c7a46e4c16b799827a7063df5cdcc5a20a8665c7702897421fb1345bcd70b
Oct 30 19:27:42.356131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.356053    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:42.356227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.356170    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:42.356227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.356200    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:42.812254 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.812201    6777 patch_prober.go:29] interesting pod/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:27:42.812808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.812276    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:42.849321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.849266    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" event={"ID":"fcdde26a-de3b-4a41-812a-cefa04196839","Type":"ContainerStarted","Data":"50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a"}
Oct 30 19:27:42.849522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.849345    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" event={"ID":"fcdde26a-de3b-4a41-812a-cefa04196839","Type":"ContainerStarted","Data":"108c7a46e4c16b799827a7063df5cdcc5a20a8665c7702897421fb1345bcd70b"}
Oct 30 19:27:42.884018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:42.883942    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" podStartSLOduration=1.8839228110000001 podStartE2EDuration="1.883922811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:42.878384561 +0000 UTC m=+1244.875299947" watchObservedRunningTime="2026-10-30 19:27:42.883922811 +0000 UTC m=+1244.880838201"
Oct 30 19:27:43.023362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:43.023315    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:43.025180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:43.025147    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:27:43.025313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:43.025209    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 19:27:43.741821 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:27:43.740832    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:27:44.023365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:44.023278    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:27:44.023365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:44.023332    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 19:27:45.023165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:45.023121    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:27:45.023355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:45.023182    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 19:27:46.061774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:46.061735    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:27:46.064962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:46.064915    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 19:27:47.023321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.023288    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:27:47.023541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.023349    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 19:27:47.078105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.078052    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"]
Oct 30 19:27:47.082337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.082312    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-254077ba-frb4v"]
Oct 30 19:27:47.933860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.933821    6777 generic.go:356] "Generic (PLEG): container finished" podID="cb33245b-7109-4488-8a23-9633a63c42dc" containerID="00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e" exitCode=0
Oct 30 19:27:47.934123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.933893    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" event={"ID":"cb33245b-7109-4488-8a23-9633a63c42dc","Type":"ContainerDied","Data":"00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e"}
Oct 30 19:27:47.934434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.934419    6777 scope.go:122] "RemoveContainer" containerID="00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e"
Oct 30 19:27:47.936386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.936361    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:47.936469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.936427    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:47.936508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:47.936473    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:27:48.023388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:48.023287    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:27:48.023388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:48.023345    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 19:27:48.130288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:48.129881    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2644a8-ce28-4367-830d-6e0bc869a22f" path="/var/lib/kubelet/pods/9d2644a8-ce28-4367-830d-6e0bc869a22f/volumes"
Oct 30 19:27:48.946844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:48.946799    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" event={"ID":"cb33245b-7109-4488-8a23-9633a63c42dc","Type":"ContainerStarted","Data":"615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018"}
Oct 30 19:27:49.023412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:49.023376    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:27:49.023686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:49.023431    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 19:27:49.344175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:49.344138    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:27:49.344664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:49.344330    6777 patch_prober.go:29] interesting pod/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:27:49.344664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:49.344383    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:27:50.028339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:50.028305    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:50.028817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:50.028791    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:50.029578 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:50.029559    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:27:50.344665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:50.344616    6777 patch_prober.go:29] interesting pod/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:27:50.345240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:50.344679    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:27:51.344011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:51.343970    6777 patch_prober.go:29] interesting pod/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:27:51.344364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:51.344333    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:27:51.839842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:51.839814    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:27:51.840278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:51.839880    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:27:52.344088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:52.344040    6777 patch_prober.go:29] interesting pod/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:27:52.344271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:52.344134    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:27:53.344005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:53.343971    6777 patch_prober.go:29] interesting pod/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:27:53.344436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:53.344039    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 19:27:54.424448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:54.424413    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:27:54.425484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:54.425451    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:27:54.426634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:54.426607    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:27:58.819822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:58.819788    6777 scope.go:122] "RemoveContainer" containerID="455dfb379cbe260b91c41cd84efadad900de33923a70d712d9d7d3e95703971d"
Oct 30 19:27:58.834722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:58.834697    6777 scope.go:122] "RemoveContainer" containerID="21fe38cf5d4dfe5836d8dd6db8f0c184bad78e311f78ee0c9b5ebc1afb116733"
Oct 30 19:27:58.850175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:58.850146    6777 scope.go:122] "RemoveContainer" containerID="b57cc5b6d317367006f208849397dc341edcced02f947ae7a4b7d2a08fc1d2c2"
Oct 30 19:27:58.862394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:58.862367    6777 scope.go:122] "RemoveContainer" containerID="8aa953bfe0f9c425166ec64e0a2aa0316b038f77f9b0bc06a48b4a5a05446bd8"
Oct 30 19:27:58.872888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:58.872865    6777 scope.go:122] "RemoveContainer" containerID="b2574e8b7b21d74a9adbcf57c19ddbd2f6d03ce0096623344d25c96c506e49e5"
Oct 30 19:27:58.882925 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:58.882901    6777 scope.go:122] "RemoveContainer" containerID="f46093d6fc3b374b9669e6caa5736b86bb963c9ce8003f23444c44febc3e1a57"
Oct 30 19:27:59.043849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:59.043820    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"]
Oct 30 19:27:59.048365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:27:59.048335    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ae66ae-lf49h"]
Oct 30 19:28:00.114841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:00.114806    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3734c02f-d420-48f8-8029-75540531648b" path="/var/lib/kubelet/pods/3734c02f-d420-48f8-8029-75540531648b/volumes"
Oct 30 19:28:07.976489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:07.976447    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:28:12.106312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:12.080224    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"]
Oct 30 19:28:12.121551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:12.121513    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a1d276-kt4v5"]
Oct 30 19:28:13.642506 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:28:13.642466    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:28:14.115131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:14.115094    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0693cb-88af-47d6-adf3-083d4466a04c" path="/var/lib/kubelet/pods/1d0693cb-88af-47d6-adf3-083d4466a04c/volumes"
Oct 30 19:28:14.817194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:14.817155    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"]
Oct 30 19:28:14.829000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:14.828970    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:14.877085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:14.861390    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f4012a-openshift-default-dockercfg-58c7t\""
Oct 30 19:28:14.941151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:14.940179    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"]
Oct 30 19:28:15.434782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.434739    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-data\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.434992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.434861    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-token\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.434992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.434941    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istiod-ca-cert\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.435156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.435015    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926mt\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-kube-api-access-926mt\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.435156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.435075    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-certs\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.435156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.435112    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-envoy\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.435328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.435257    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-socket\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.435328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.435323    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-credential-socket\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.435429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.435364    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-podinfo\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.537968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.537926    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-credential-socket\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.538185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.537993    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-podinfo\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.538185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.538152    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-data\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.538321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.538258    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-token\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.538385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.538350    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istiod-ca-cert\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.538442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.538416    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926mt\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-kube-api-access-926mt\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.538504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.538453    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-certs\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.538504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.538488    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-envoy\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.538667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.538649    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-socket\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.539325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.539294    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-socket\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.539595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.539574    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-certs\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.540113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.540076    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istiod-ca-cert\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.540747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.540723    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-credential-socket\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.541217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.541192    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-data\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.626226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.615033    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-podinfo\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.644085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.636721    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-926mt\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-kube-api-access-926mt\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.778133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.760211    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-envoy\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.850634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.850574    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"]
Oct 30 19:28:15.890351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.890315    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-token\") pod \"e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") " pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:15.978453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:15.978418    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:16.103150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.094417    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:16.138366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.138331    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"]
Oct 30 19:28:16.168567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.153374    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8ceda0\""
Oct 30 19:28:16.168567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.153482    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8ceda0-dockercfg-8kwt4\""
Oct 30 19:28:16.266083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.263740    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86652ae5-47e8-49f3-afd6-4c75416ce45b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c\" (UID: \"86652ae5-47e8-49f3-afd6-4c75416ce45b\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:16.266083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.263816    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6prl\" (UniqueName: \"kubernetes.io/projected/86652ae5-47e8-49f3-afd6-4c75416ce45b-kube-api-access-g6prl\") pod \"payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c\" (UID: \"86652ae5-47e8-49f3-afd6-4c75416ce45b\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:16.378138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.377229    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86652ae5-47e8-49f3-afd6-4c75416ce45b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c\" (UID: \"86652ae5-47e8-49f3-afd6-4c75416ce45b\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:16.378138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.377299    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6prl\" (UniqueName: \"kubernetes.io/projected/86652ae5-47e8-49f3-afd6-4c75416ce45b-kube-api-access-g6prl\") pod \"payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c\" (UID: \"86652ae5-47e8-49f3-afd6-4c75416ce45b\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:16.403114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.378852    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86652ae5-47e8-49f3-afd6-4c75416ce45b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c\" (UID: \"86652ae5-47e8-49f3-afd6-4c75416ce45b\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:16.420725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.420551    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6prl\" (UniqueName: \"kubernetes.io/projected/86652ae5-47e8-49f3-afd6-4c75416ce45b-kube-api-access-g6prl\") pod \"payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c\" (UID: \"86652ae5-47e8-49f3-afd6-4c75416ce45b\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:16.609081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.608764    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:16.716277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.716236    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"]
Oct 30 19:28:16.748649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.748342    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:16.748896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.748874    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:16.749042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:16.749026    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:17.026134 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:28:17.026103    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86652ae5_47e8_49f3_afd6_4c75416ce45b.slice/crio-a79b78eea267bbaaa5f6d7f1b46ea5f07e28e4f7e2c92b2d7bd96e7d783abead WatchSource:0}: Error finding container a79b78eea267bbaaa5f6d7f1b46ea5f07e28e4f7e2c92b2d7bd96e7d783abead: Status 404 returned error can't find the container with id a79b78eea267bbaaa5f6d7f1b46ea5f07e28e4f7e2c92b2d7bd96e7d783abead
Oct 30 19:28:17.026511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:17.026487    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"]
Oct 30 19:28:17.732620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:17.732579    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" event={"ID":"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78","Type":"ContainerStarted","Data":"5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db"}
Oct 30 19:28:17.732904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:17.732654    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" event={"ID":"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78","Type":"ContainerStarted","Data":"abf995ff3890c77adb4214d1207652d877324c0429e1ad7df945a6ea4b4017fa"}
Oct 30 19:28:17.736354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:17.736234    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c" event={"ID":"86652ae5-47e8-49f3-afd6-4c75416ce45b","Type":"ContainerStarted","Data":"ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e"}
Oct 30 19:28:17.736354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:17.736288    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c" event={"ID":"86652ae5-47e8-49f3-afd6-4c75416ce45b","Type":"ContainerStarted","Data":"a79b78eea267bbaaa5f6d7f1b46ea5f07e28e4f7e2c92b2d7bd96e7d783abead"}
Oct 30 19:28:17.736590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:17.736429    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:17.760736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:17.760682    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podStartSLOduration=3.760664061 podStartE2EDuration="3.760664061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:17.758272601 +0000 UTC m=+1279.755187987" watchObservedRunningTime="2026-10-30 19:28:17.760664061 +0000 UTC m=+1279.757579447"
Oct 30 19:28:17.785457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:17.785400    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c" podStartSLOduration=2.785385623 podStartE2EDuration="2.785385623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:17.780773593 +0000 UTC m=+1279.777688979" watchObservedRunningTime="2026-10-30 19:28:17.785385623 +0000 UTC m=+1279.782301008"
Oct 30 19:28:18.012880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:18.012788    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:18.021653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:18.021625    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:18.745988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:18.745939    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"]
Oct 30 19:28:18.754294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:18.754256    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:18.755488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:18.755461    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:28:19.617634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:19.617241    6777 patch_prober.go:29] interesting pod/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:19.617634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:19.617319    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:22.034802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.034755    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"]
Oct 30 19:28:22.093602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.093564    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"]
Oct 30 19:28:22.094721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.094133    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"]
Oct 30 19:28:22.094868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.094814    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:22.102514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.102482    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f4012a\""
Oct 30 19:28:22.110714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.108464    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:22.116893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.115763    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f4012a-dockercfg-qbkp7\""
Oct 30 19:28:22.133474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.133444    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"]
Oct 30 19:28:22.270086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.265422    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m\" (UID: \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:22.270086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.265730    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbxqx\" (UniqueName: \"kubernetes.io/projected/8059c66f-eff9-4f08-99a3-9c216aa17183-kube-api-access-kbxqx\") pod \"payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w\" (UID: \"8059c66f-eff9-4f08-99a3-9c216aa17183\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:22.270086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.266099    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8059c66f-eff9-4f08-99a3-9c216aa17183-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w\" (UID: \"8059c66f-eff9-4f08-99a3-9c216aa17183\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:22.270086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.266151    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcl8\" (UniqueName: \"kubernetes.io/projected/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-kube-api-access-bjcl8\") pod \"payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m\" (UID: \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:22.503163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.434350    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8059c66f-eff9-4f08-99a3-9c216aa17183-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w\" (UID: \"8059c66f-eff9-4f08-99a3-9c216aa17183\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:22.503163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.437594    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcl8\" (UniqueName: \"kubernetes.io/projected/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-kube-api-access-bjcl8\") pod \"payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m\" (UID: \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:22.503163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.437919    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m\" (UID: \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:22.503163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.438028    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbxqx\" (UniqueName: \"kubernetes.io/projected/8059c66f-eff9-4f08-99a3-9c216aa17183-kube-api-access-kbxqx\") pod \"payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w\" (UID: \"8059c66f-eff9-4f08-99a3-9c216aa17183\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:22.503163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.439573    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8059c66f-eff9-4f08-99a3-9c216aa17183-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w\" (UID: \"8059c66f-eff9-4f08-99a3-9c216aa17183\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:22.503163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.443469    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m\" (UID: \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:22.503163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.476971    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbxqx\" (UniqueName: \"kubernetes.io/projected/8059c66f-eff9-4f08-99a3-9c216aa17183-kube-api-access-kbxqx\") pod \"payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w\" (UID: \"8059c66f-eff9-4f08-99a3-9c216aa17183\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:22.530144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.530107    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcl8\" (UniqueName: \"kubernetes.io/projected/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-kube-api-access-bjcl8\") pod \"payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m\" (UID: \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:22.734884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.734838    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:22.745987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:22.745953    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:23.098127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.098084    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"]
Oct 30 19:28:23.100123 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:28:23.100080    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8059c66f_eff9_4f08_99a3_9c216aa17183.slice/crio-f46074fdae3e0d4094080cc8ee6445b2ac482d5c47dc801a90edaaa061b7fac6 WatchSource:0}: Error finding container f46074fdae3e0d4094080cc8ee6445b2ac482d5c47dc801a90edaaa061b7fac6: Status 404 returned error can't find the container with id f46074fdae3e0d4094080cc8ee6445b2ac482d5c47dc801a90edaaa061b7fac6
Oct 30 19:28:23.304433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.304400    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"]
Oct 30 19:28:23.305431 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:28:23.305405    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6f67de_1e81_4bc3_9489_d2a24bbba61a.slice/crio-1480e9903f1f1b8d1f7f5a6e3a1a59858e1b79d2ddf41e33f12b08b48bc74dfe WatchSource:0}: Error finding container 1480e9903f1f1b8d1f7f5a6e3a1a59858e1b79d2ddf41e33f12b08b48bc74dfe: Status 404 returned error can't find the container with id 1480e9903f1f1b8d1f7f5a6e3a1a59858e1b79d2ddf41e33f12b08b48bc74dfe
Oct 30 19:28:23.753167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.751261    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45" podUID="ef1ef741-11df-4980-9405-c51808b647bc" containerName="payload-pre-processing" containerID="cri-o://604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f" gracePeriod=25
Oct 30 19:28:23.814871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.814831    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m" event={"ID":"ff6f67de-1e81-4bc3-9489-d2a24bbba61a","Type":"ContainerStarted","Data":"4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372"}
Oct 30 19:28:23.815135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.814901    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m" event={"ID":"ff6f67de-1e81-4bc3-9489-d2a24bbba61a","Type":"ContainerStarted","Data":"1480e9903f1f1b8d1f7f5a6e3a1a59858e1b79d2ddf41e33f12b08b48bc74dfe"}
Oct 30 19:28:23.815135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.814947    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:23.817323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.817289    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w" event={"ID":"8059c66f-eff9-4f08-99a3-9c216aa17183","Type":"ContainerStarted","Data":"9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9"}
Oct 30 19:28:23.817456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.817346    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w" event={"ID":"8059c66f-eff9-4f08-99a3-9c216aa17183","Type":"ContainerStarted","Data":"f46074fdae3e0d4094080cc8ee6445b2ac482d5c47dc801a90edaaa061b7fac6"}
Oct 30 19:28:23.817456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.817379    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:23.837920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.835708    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m" podStartSLOduration=1.835688496 podStartE2EDuration="1.835688496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:23.834447321 +0000 UTC m=+1285.831362709" watchObservedRunningTime="2026-10-30 19:28:23.835688496 +0000 UTC m=+1285.832603883"
Oct 30 19:28:23.869804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:23.869725    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w" podStartSLOduration=1.869700732 podStartE2EDuration="1.869700732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:23.859198651 +0000 UTC m=+1285.856114037" watchObservedRunningTime="2026-10-30 19:28:23.869700732 +0000 UTC m=+1285.866616118"
Oct 30 19:28:24.217243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.217208    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:28:24.290766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.290724    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1ef741-11df-4980-9405-c51808b647bc-kube-api-access-ptb2j\" (UniqueName: \"kubernetes.io/projected/ef1ef741-11df-4980-9405-c51808b647bc-kube-api-access-ptb2j\") pod \"ef1ef741-11df-4980-9405-c51808b647bc\" (UID: \"ef1ef741-11df-4980-9405-c51808b647bc\") "
Oct 30 19:28:24.291038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.291020    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1ef741-11df-4980-9405-c51808b647bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ef741-11df-4980-9405-c51808b647bc-plugins-config-volume\") pod \"ef1ef741-11df-4980-9405-c51808b647bc\" (UID: \"ef1ef741-11df-4980-9405-c51808b647bc\") "
Oct 30 19:28:24.295550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.293164    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1ef741-11df-4980-9405-c51808b647bc-plugins-config-volume" pod "ef1ef741-11df-4980-9405-c51808b647bc" (UID: "ef1ef741-11df-4980-9405-c51808b647bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:24.295550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.294326    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1ef741-11df-4980-9405-c51808b647bc-kube-api-access-ptb2j" pod "ef1ef741-11df-4980-9405-c51808b647bc" (UID: "ef1ef741-11df-4980-9405-c51808b647bc"). InnerVolumeSpecName "kube-api-access-ptb2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:24.394562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.394521    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptb2j\" (UniqueName: \"kubernetes.io/projected/ef1ef741-11df-4980-9405-c51808b647bc-kube-api-access-ptb2j\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:24.394562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.394557    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1ef741-11df-4980-9405-c51808b647bc-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:24.832828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.832147    6777 generic.go:356] "Generic (PLEG): container finished" podID="ef1ef741-11df-4980-9405-c51808b647bc" containerID="604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f" exitCode=0
Oct 30 19:28:24.852347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.833524    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"
Oct 30 19:28:24.852347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.837745    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45" event={"ID":"ef1ef741-11df-4980-9405-c51808b647bc","Type":"ContainerDied","Data":"604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f"}
Oct 30 19:28:24.852347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.837844    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45" event={"ID":"ef1ef741-11df-4980-9405-c51808b647bc","Type":"ContainerDied","Data":"68aac96cc0ba959d0a5f366c08da8eee944be79fa1c30140e1a06eaf87769502"}
Oct 30 19:28:24.852347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.837882    6777 scope.go:122] "RemoveContainer" containerID="604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f"
Oct 30 19:28:24.994081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:24.991428    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"]
Oct 30 19:28:25.055084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:25.052861    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a718a-67df85b967-xbx45"]
Oct 30 19:28:25.111355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:25.108258    6777 scope.go:122] "RemoveContainer" containerID="604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f"
Oct 30 19:28:25.111355 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:28:25.108704    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f\": container with ID starting with 604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f not found: ID does not exist" containerID="604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f"
Oct 30 19:28:25.111355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:25.108740    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f"} err="failed to get container status \"604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f\": rpc error: code = NotFound desc = could not find container \"604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f\": container with ID starting with 604d5cf21327c1348fa92d36253e8b7d18fb75aa1bbf0705a09e7d0e2121cd7f not found: ID does not exist"
Oct 30 19:28:26.114437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.114405    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1ef741-11df-4980-9405-c51808b647bc" path="/var/lib/kubelet/pods/ef1ef741-11df-4980-9405-c51808b647bc/volumes"
Oct 30 19:28:26.865220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.865187    6777 generic.go:356] "Generic (PLEG): container finished" podID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerID="91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec" exitCode=0
Oct 30 19:28:26.865464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.865260    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" event={"ID":"81cb5dd9-9980-4623-9d2d-67f167639c95","Type":"ContainerDied","Data":"91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec"}
Oct 30 19:28:26.865824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.865803    6777 scope.go:122] "RemoveContainer" containerID="91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec"
Oct 30 19:28:26.867440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.867420    6777 generic.go:356] "Generic (PLEG): container finished" podID="969296cc-bcee-4ad7-8944-977164eac29c" containerID="c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca" exitCode=0
Oct 30 19:28:26.867525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.867483    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" event={"ID":"969296cc-bcee-4ad7-8944-977164eac29c","Type":"ContainerDied","Data":"c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca"}
Oct 30 19:28:26.867874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.867861    6777 scope.go:122] "RemoveContainer" containerID="c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca"
Oct 30 19:28:26.867927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.867901    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:26.867982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.867970    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:26.868019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.868004    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:26.869653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.869627    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:26.869727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.869698    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:26.869764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:26.869730    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:27.439233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:27.439204    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:28:27.929728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:27.929690    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" event={"ID":"969296cc-bcee-4ad7-8944-977164eac29c","Type":"ContainerStarted","Data":"e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847"}
Oct 30 19:28:27.932077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:27.932032    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" event={"ID":"81cb5dd9-9980-4623-9d2d-67f167639c95","Type":"ContainerStarted","Data":"1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c"}
Oct 30 19:28:28.473090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:28.472980    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:28:28.473609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:28.473191    6777 patch_prober.go:29] interesting pod/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:28:28.473609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:28.473261    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 19:28:28.716736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:28.716581    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:28:28.716736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:28.716717    6777 patch_prober.go:29] interesting pod/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:28:28.717016 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:28.716771    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 19:28:28.799424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:28.799380    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:28:29.486213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:29.486162    6777 patch_prober.go:29] interesting pod/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:28:29.554107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:29.535144    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 19:28:29.756136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:29.720219    6777 patch_prober.go:29] interesting pod/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:28:29.756526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:29.756487    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 19:28:30.499080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.498240    6777 patch_prober.go:29] interesting pod/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:28:30.499080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.498307    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 19:28:30.604242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.603749    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"]
Oct 30 19:28:30.605858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.605829    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ef1ef741-11df-4980-9405-c51808b647bc" containerName="payload-pre-processing"
Oct 30 19:28:30.606024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.606007    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1ef741-11df-4980-9405-c51808b647bc" containerName="payload-pre-processing"
Oct 30 19:28:30.606335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.606316    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1ef741-11df-4980-9405-c51808b647bc" containerName="payload-pre-processing"
Oct 30 19:28:30.610049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.610033    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:30.611707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.611679    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"]
Oct 30 19:28:30.614690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.614673    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:30.615681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.615666    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:28:30.726080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.688591    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b5f10ff-grnps\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:30.729371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.729337    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmhw\" (UniqueName: \"kubernetes.io/projected/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-kube-api-access-thmhw\") pod \"maas-api-revoke-sub-9b5f10ff-grnps\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:30.752146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.733510    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b5f10ff-grnps\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:30.752738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.729079    6777 patch_prober.go:29] interesting pod/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:28:30.752974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.752942    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 19:28:30.878084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.870701    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmhw\" (UniqueName: \"kubernetes.io/projected/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-kube-api-access-thmhw\") pod \"maas-api-revoke-sub-9b5f10ff-grnps\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:30.892389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.892343    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b5f10ff-grnps\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:30.893284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.893258    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b5f10ff-grnps\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:30.944263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.944226    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b5f10ff-grnps\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:30.944859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:30.944785    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b5f10ff-grnps\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:31.037371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:31.028450    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmhw\" (UniqueName: \"kubernetes.io/projected/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-kube-api-access-thmhw\") pod \"maas-api-revoke-sub-9b5f10ff-grnps\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:31.237084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:31.236561    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:31.423162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:31.423100    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"]
Oct 30 19:28:31.425809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:31.424553    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy" containerID="cri-o://468cad87600f5831a09e192c862d1152e8701b5035c6194dd144b07cf3fb8d88" gracePeriod=30
Oct 30 19:28:31.690147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:31.689820    6777 patch_prober.go:29] interesting pod/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:28:31.690147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:31.689898    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 19:28:31.723906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:31.719706    6777 patch_prober.go:29] interesting pod/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:28:31.723906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:31.719784    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 19:28:32.321443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:32.321408    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"]
Oct 30 19:28:32.408050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:32.407956    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps" event={"ID":"b02f4f5e-8d7e-412e-9b05-dc2893af31d1","Type":"ContainerStarted","Data":"9fb5763d47c46796dd53b4126493a34c33cf056b2ba8621c7d92dca1cb120c60"}
Oct 30 19:28:32.474146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:32.472770    6777 patch_prober.go:29] interesting pod/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:28:32.474146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:32.472841    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 19:28:32.715525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:32.715488    6777 patch_prober.go:29] interesting pod/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:28:32.715921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:32.715546    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 19:28:33.422893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:33.422853    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps" event={"ID":"b02f4f5e-8d7e-412e-9b05-dc2893af31d1","Type":"ContainerStarted","Data":"3b6841fb24347d71448857bb4d299ed136295b515864c59adbc4f962f6970a72"}
Oct 30 19:28:33.488087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:33.482240    6777 patch_prober.go:29] interesting pod/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:28:33.488087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:33.482305    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 19:28:33.582091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:33.578272    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:28:33.715894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:33.715788    6777 patch_prober.go:29] interesting pod/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:28:33.715894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:33.715869    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 19:28:34.549091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:34.544235    6777 generic.go:356] "Generic (PLEG): container finished" podID="b02f4f5e-8d7e-412e-9b05-dc2893af31d1" containerID="3b6841fb24347d71448857bb4d299ed136295b515864c59adbc4f962f6970a72" exitCode=0
Oct 30 19:28:34.549091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:34.544605    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps" event={"ID":"b02f4f5e-8d7e-412e-9b05-dc2893af31d1","Type":"ContainerDied","Data":"3b6841fb24347d71448857bb4d299ed136295b515864c59adbc4f962f6970a72"}
Oct 30 19:28:34.724640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:34.724601    6777 patch_prober.go:29] interesting pod/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:28:34.731164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:34.725302    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 19:28:34.849708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:34.849576    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:28:34.870443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:34.867018    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:28:34.966083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:34.965192    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:28:34.979346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:34.979237    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:28:35.086533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.086497    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"]
Oct 30 19:28:35.090978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.090794    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:35.119459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.110428    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-33b4d7-dockercfg-2msxc\""
Oct 30 19:28:35.119459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.113814    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-33b4d7\""
Oct 30 19:28:35.120394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.120371    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"]
Oct 30 19:28:35.140689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.140655    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"]
Oct 30 19:28:35.146269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.146238    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:35.166957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.166926    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"]
Oct 30 19:28:35.185116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.185076    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp\" (UID: \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:35.185300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.185191    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhrv\" (UniqueName: \"kubernetes.io/projected/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-kube-api-access-ldhrv\") pod \"payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp\" (UID: \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:35.340160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.340038    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef5a740-4ced-428e-bb19-9092a9ae471f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw\" (UID: \"fef5a740-4ced-428e-bb19-9092a9ae471f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:35.359890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.359791    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcw8r\" (UniqueName: \"kubernetes.io/projected/fef5a740-4ced-428e-bb19-9092a9ae471f-kube-api-access-dcw8r\") pod \"payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw\" (UID: \"fef5a740-4ced-428e-bb19-9092a9ae471f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:35.370085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.360327    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp\" (UID: \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:35.384164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.383851    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp\" (UID: \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:35.389593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.389554    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhrv\" (UniqueName: \"kubernetes.io/projected/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-kube-api-access-ldhrv\") pod \"payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp\" (UID: \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:35.533094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.529998    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhrv\" (UniqueName: \"kubernetes.io/projected/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-kube-api-access-ldhrv\") pod \"payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp\" (UID: \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:35.533094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.531554    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcw8r\" (UniqueName: \"kubernetes.io/projected/fef5a740-4ced-428e-bb19-9092a9ae471f-kube-api-access-dcw8r\") pod \"payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw\" (UID: \"fef5a740-4ced-428e-bb19-9092a9ae471f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:35.533094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.531955    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef5a740-4ced-428e-bb19-9092a9ae471f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw\" (UID: \"fef5a740-4ced-428e-bb19-9092a9ae471f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:35.533900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.533102    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef5a740-4ced-428e-bb19-9092a9ae471f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw\" (UID: \"fef5a740-4ced-428e-bb19-9092a9ae471f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:35.575937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.567721    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcw8r\" (UniqueName: \"kubernetes.io/projected/fef5a740-4ced-428e-bb19-9092a9ae471f-kube-api-access-dcw8r\") pod \"payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw\" (UID: \"fef5a740-4ced-428e-bb19-9092a9ae471f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:35.720324 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.720292    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:28:35.720853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.720825    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:28:35.721600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.721578    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:28:35.724856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.724836    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:35.766421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.766390    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:35.785521 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.785495    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:35.838739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.837390    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-openshift-ca\") pod \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") "
Oct 30 19:28:35.838739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.837563    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-xks-ca\") pod \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") "
Oct 30 19:28:35.838739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.837841    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-kube-api-access-thmhw\" (UniqueName: \"kubernetes.io/projected/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-kube-api-access-thmhw\") pod \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\" (UID: \"b02f4f5e-8d7e-412e-9b05-dc2893af31d1\") "
Oct 30 19:28:35.839359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.839312    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-openshift-ca" pod "b02f4f5e-8d7e-412e-9b05-dc2893af31d1" (UID: "b02f4f5e-8d7e-412e-9b05-dc2893af31d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:35.842656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.842623    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-kube-api-access-thmhw" pod "b02f4f5e-8d7e-412e-9b05-dc2893af31d1" (UID: "b02f4f5e-8d7e-412e-9b05-dc2893af31d1"). InnerVolumeSpecName "kube-api-access-thmhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:35.843258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.843232    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-xks-ca" pod "b02f4f5e-8d7e-412e-9b05-dc2893af31d1" (UID: "b02f4f5e-8d7e-412e-9b05-dc2893af31d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:35.942805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.942369    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thmhw\" (UniqueName: \"kubernetes.io/projected/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-kube-api-access-thmhw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:35.942805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.942406    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:35.942805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:35.942424    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02f4f5e-8d7e-412e-9b05-dc2893af31d1-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.020445 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:28:36.020414    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef5a740_4ced_428e_bb19_9092a9ae471f.slice/crio-d8b5adea301c82de6a7079fa6ce3e2c962cc1110a51160cbba32b52f33c55868 WatchSource:0}: Error finding container d8b5adea301c82de6a7079fa6ce3e2c962cc1110a51160cbba32b52f33c55868: Status 404 returned error can't find the container with id d8b5adea301c82de6a7079fa6ce3e2c962cc1110a51160cbba32b52f33c55868
Oct 30 19:28:36.021213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.021182    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"]
Oct 30 19:28:36.103620 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:28:36.103592    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc79b5d0_dbf8_413d_ad18_216d07d3ad08.slice/crio-53cef52c45a073ae32ca382d3746c316899912749e73d46831c86209ebf16981 WatchSource:0}: Error finding container 53cef52c45a073ae32ca382d3746c316899912749e73d46831c86209ebf16981: Status 404 returned error can't find the container with id 53cef52c45a073ae32ca382d3746c316899912749e73d46831c86209ebf16981
Oct 30 19:28:36.119004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.118780    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"]
Oct 30 19:28:36.610036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.609530    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"
Oct 30 19:28:36.610036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.609529    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps" event={"ID":"b02f4f5e-8d7e-412e-9b05-dc2893af31d1","Type":"ContainerDied","Data":"9fb5763d47c46796dd53b4126493a34c33cf056b2ba8621c7d92dca1cb120c60"}
Oct 30 19:28:36.610036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.609666    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fb5763d47c46796dd53b4126493a34c33cf056b2ba8621c7d92dca1cb120c60"
Oct 30 19:28:36.612284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.612258    6777 generic.go:356] "Generic (PLEG): container finished" podID="280439f1-4695-45a6-b321-6f65f88cc8df" containerID="468cad87600f5831a09e192c862d1152e8701b5035c6194dd144b07cf3fb8d88" exitCode=0
Oct 30 19:28:36.612420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.612330    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" event={"ID":"280439f1-4695-45a6-b321-6f65f88cc8df","Type":"ContainerDied","Data":"468cad87600f5831a09e192c862d1152e8701b5035c6194dd144b07cf3fb8d88"}
Oct 30 19:28:36.612420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.612390    6777 scope.go:122] "RemoveContainer" containerID="ee6e4e263af6b3c180b1b00c1c181b6612787f850251ec4f4aae9738a5eea846"
Oct 30 19:28:36.615445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.615360    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw" event={"ID":"fef5a740-4ced-428e-bb19-9092a9ae471f","Type":"ContainerStarted","Data":"f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2"}
Oct 30 19:28:36.615445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.615412    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw" event={"ID":"fef5a740-4ced-428e-bb19-9092a9ae471f","Type":"ContainerStarted","Data":"d8b5adea301c82de6a7079fa6ce3e2c962cc1110a51160cbba32b52f33c55868"}
Oct 30 19:28:36.616803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.616439    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:36.619227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.619200    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp" event={"ID":"cc79b5d0-dbf8-413d-ad18-216d07d3ad08","Type":"ContainerStarted","Data":"e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5"}
Oct 30 19:28:36.619337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.619309    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp" event={"ID":"cc79b5d0-dbf8-413d-ad18-216d07d3ad08","Type":"ContainerStarted","Data":"53cef52c45a073ae32ca382d3746c316899912749e73d46831c86209ebf16981"}
Oct 30 19:28:36.619961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.619945    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:36.652667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.652637    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:28:36.655015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.654972    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw" podStartSLOduration=1.654950468 podStartE2EDuration="1.654950468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:36.649271518 +0000 UTC m=+1298.646186904" watchObservedRunningTime="2026-10-30 19:28:36.654950468 +0000 UTC m=+1298.651865853"
Oct 30 19:28:36.713007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.712902    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp" podStartSLOduration=1.712888919 podStartE2EDuration="1.712888919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:36.710307073 +0000 UTC m=+1298.707222461" watchObservedRunningTime="2026-10-30 19:28:36.712888919 +0000 UTC m=+1298.709804304"
Oct 30 19:28:36.768971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.768935    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-credential-socket\") pod \"280439f1-4695-45a6-b321-6f65f88cc8df\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") "
Oct 30 19:28:36.769501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.768987    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-data\") pod \"280439f1-4695-45a6-b321-6f65f88cc8df\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") "
Oct 30 19:28:36.769501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769040    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-certs\") pod \"280439f1-4695-45a6-b321-6f65f88cc8df\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") "
Oct 30 19:28:36.769501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769119    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-envoy\") pod \"280439f1-4695-45a6-b321-6f65f88cc8df\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") "
Oct 30 19:28:36.769501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769163    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/280439f1-4695-45a6-b321-6f65f88cc8df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280439f1-4695-45a6-b321-6f65f88cc8df-istio-podinfo\") pod \"280439f1-4695-45a6-b321-6f65f88cc8df\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") "
Oct 30 19:28:36.769501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769266    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-data" pod "280439f1-4695-45a6-b321-6f65f88cc8df" (UID: "280439f1-4695-45a6-b321-6f65f88cc8df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:36.769501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769298    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-credential-socket" pod "280439f1-4695-45a6-b321-6f65f88cc8df" (UID: "280439f1-4695-45a6-b321-6f65f88cc8df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:36.769501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769352    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-istio-token\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-istio-token\") pod \"280439f1-4695-45a6-b321-6f65f88cc8df\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") "
Oct 30 19:28:36.769501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769446    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280439f1-4695-45a6-b321-6f65f88cc8df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280439f1-4695-45a6-b321-6f65f88cc8df-istiod-ca-cert\") pod \"280439f1-4695-45a6-b321-6f65f88cc8df\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") "
Oct 30 19:28:36.769895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769539    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-kube-api-access-pbnlm\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-kube-api-access-pbnlm\") pod \"280439f1-4695-45a6-b321-6f65f88cc8df\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") "
Oct 30 19:28:36.769895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769607    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-socket\") pod \"280439f1-4695-45a6-b321-6f65f88cc8df\" (UID: \"280439f1-4695-45a6-b321-6f65f88cc8df\") "
Oct 30 19:28:36.769895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769844    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-certs" pod "280439f1-4695-45a6-b321-6f65f88cc8df" (UID: "280439f1-4695-45a6-b321-6f65f88cc8df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:36.770048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.769942    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280439f1-4695-45a6-b321-6f65f88cc8df-istiod-ca-cert" pod "280439f1-4695-45a6-b321-6f65f88cc8df" (UID: "280439f1-4695-45a6-b321-6f65f88cc8df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:36.770131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.770111    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-socket" pod "280439f1-4695-45a6-b321-6f65f88cc8df" (UID: "280439f1-4695-45a6-b321-6f65f88cc8df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:36.770545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.770515    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.770545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.770542    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.770724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.770557    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.770724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.770571    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280439f1-4695-45a6-b321-6f65f88cc8df-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.770724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.770585    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.772858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.772726    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/280439f1-4695-45a6-b321-6f65f88cc8df-istio-podinfo" pod "280439f1-4695-45a6-b321-6f65f88cc8df" (UID: "280439f1-4695-45a6-b321-6f65f88cc8df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:28:36.772858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.772742    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-istio-token" pod "280439f1-4695-45a6-b321-6f65f88cc8df" (UID: "280439f1-4695-45a6-b321-6f65f88cc8df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:36.772858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.772828    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-envoy" pod "280439f1-4695-45a6-b321-6f65f88cc8df" (UID: "280439f1-4695-45a6-b321-6f65f88cc8df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:36.773090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.773036    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-kube-api-access-pbnlm" pod "280439f1-4695-45a6-b321-6f65f88cc8df" (UID: "280439f1-4695-45a6-b321-6f65f88cc8df"). InnerVolumeSpecName "kube-api-access-pbnlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:36.874111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.873803    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbnlm\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-kube-api-access-pbnlm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.874111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.873920    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280439f1-4695-45a6-b321-6f65f88cc8df-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.874111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.873940    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280439f1-4695-45a6-b321-6f65f88cc8df-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.874111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:36.873956    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280439f1-4695-45a6-b321-6f65f88cc8df-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:37.049210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:37.049163    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"]
Oct 30 19:28:37.057292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:37.057255    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a10c6ec-nnjbk"]
Oct 30 19:28:37.628401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:37.628368    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"
Oct 30 19:28:37.628589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:37.628427    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z" event={"ID":"280439f1-4695-45a6-b321-6f65f88cc8df","Type":"ContainerDied","Data":"f01d1b9a2724236af488e70b6924ae317af103caf962b3cc6480e7660d89296f"}
Oct 30 19:28:37.628589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:37.628495    6777 scope.go:122] "RemoveContainer" containerID="468cad87600f5831a09e192c862d1152e8701b5035c6194dd144b07cf3fb8d88"
Oct 30 19:28:37.673461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:37.673423    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"]
Oct 30 19:28:37.680141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:37.680111    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0d360546-openshift-default-5587bbdb7c-wnc8z"]
Oct 30 19:28:38.120929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:38.120891    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" path="/var/lib/kubelet/pods/280439f1-4695-45a6-b321-6f65f88cc8df/volumes"
Oct 30 19:28:38.121743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:38.121706    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3973d22-e097-45cc-a6d3-6fff2f5cb1da" path="/var/lib/kubelet/pods/e3973d22-e097-45cc-a6d3-6fff2f5cb1da/volumes"
Oct 30 19:28:39.059454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:39.059416    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"]
Oct 30 19:28:39.063589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:39.063564    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c1a3ae-4x46x"]
Oct 30 19:28:40.123282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.123230    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c3aee7-aa29-46d0-848b-473e5d4d46e7" path="/var/lib/kubelet/pods/48c3aee7-aa29-46d0-848b-473e5d4d46e7/volumes"
Oct 30 19:28:40.128353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.128324    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"]
Oct 30 19:28:40.129227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.129206    6777 cpu_manager.go:414] "containerMap: removing container" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy"
Oct 30 19:28:40.129227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.129223    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy"
Oct 30 19:28:40.129410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.129240    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b02f4f5e-8d7e-412e-9b05-dc2893af31d1" containerName="revoke-keys"
Oct 30 19:28:40.129410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.129246    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02f4f5e-8d7e-412e-9b05-dc2893af31d1" containerName="revoke-keys"
Oct 30 19:28:40.129491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.129447    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy"
Oct 30 19:28:40.129491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.129472    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02f4f5e-8d7e-412e-9b05-dc2893af31d1" containerName="revoke-keys"
Oct 30 19:28:40.133389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.133368    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:40.143390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.143020    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"]
Oct 30 19:28:40.144753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.144728    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-891d80\""
Oct 30 19:28:40.144986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.144968    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-891d80-dockercfg-bkcl9\""
Oct 30 19:28:40.159814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.159777    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"]
Oct 30 19:28:40.161735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.161706    6777 cpu_manager.go:414] "containerMap: removing container" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy"
Oct 30 19:28:40.161735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.161733    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy"
Oct 30 19:28:40.161989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.161973    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="280439f1-4695-45a6-b321-6f65f88cc8df" containerName="istio-proxy"
Oct 30 19:28:40.169954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.169926    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:40.175495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.175462    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"]
Oct 30 19:28:40.255080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.253271    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db3bbf8-847c-4004-8b1b-91bca311dbf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2\" (UID: \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:40.255080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.253417    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ggg\" (UniqueName: \"kubernetes.io/projected/eb1df337-9ecf-43f5-95ad-99faa700a4b0-kube-api-access-h9ggg\") pod \"payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b\" (UID: \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:40.255080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.253495    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1df337-9ecf-43f5-95ad-99faa700a4b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b\" (UID: \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:40.255080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.253564    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkk62\" (UniqueName: \"kubernetes.io/projected/1db3bbf8-847c-4004-8b1b-91bca311dbf7-kube-api-access-wkk62\") pod \"payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2\" (UID: \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:40.366318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.366279    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db3bbf8-847c-4004-8b1b-91bca311dbf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2\" (UID: \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:40.366541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.366396    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ggg\" (UniqueName: \"kubernetes.io/projected/eb1df337-9ecf-43f5-95ad-99faa700a4b0-kube-api-access-h9ggg\") pod \"payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b\" (UID: \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:40.366541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.366455    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1df337-9ecf-43f5-95ad-99faa700a4b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b\" (UID: \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:40.366541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.366521    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkk62\" (UniqueName: \"kubernetes.io/projected/1db3bbf8-847c-4004-8b1b-91bca311dbf7-kube-api-access-wkk62\") pod \"payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2\" (UID: \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:40.367622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.367597    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db3bbf8-847c-4004-8b1b-91bca311dbf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2\" (UID: \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:40.368338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.368319    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1df337-9ecf-43f5-95ad-99faa700a4b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b\" (UID: \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:40.380340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.379781    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkk62\" (UniqueName: \"kubernetes.io/projected/1db3bbf8-847c-4004-8b1b-91bca311dbf7-kube-api-access-wkk62\") pod \"payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2\" (UID: \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:40.381113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.381083    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ggg\" (UniqueName: \"kubernetes.io/projected/eb1df337-9ecf-43f5-95ad-99faa700a4b0-kube-api-access-h9ggg\") pod \"payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b\" (UID: \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:40.451398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.451363    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:40.501660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.501590    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:40.889103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.888972    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"]
Oct 30 19:28:40.889264 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:28:40.889132    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db3bbf8_847c_4004_8b1b_91bca311dbf7.slice/crio-a5d0ba2c48d35732573d6070be6f4cce7ea973e26d01d6ec098f0a7fb726fb96 WatchSource:0}: Error finding container a5d0ba2c48d35732573d6070be6f4cce7ea973e26d01d6ec098f0a7fb726fb96: Status 404 returned error can't find the container with id a5d0ba2c48d35732573d6070be6f4cce7ea973e26d01d6ec098f0a7fb726fb96
Oct 30 19:28:40.890102 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:28:40.890036    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1df337_9ecf_43f5_95ad_99faa700a4b0.slice/crio-336a80e40d25a4c27e2bb11b34f5e8517616ee506e80d3ca0a48903e737a02de WatchSource:0}: Error finding container 336a80e40d25a4c27e2bb11b34f5e8517616ee506e80d3ca0a48903e737a02de: Status 404 returned error can't find the container with id 336a80e40d25a4c27e2bb11b34f5e8517616ee506e80d3ca0a48903e737a02de
Oct 30 19:28:40.895242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:40.895222    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"]
Oct 30 19:28:41.690509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:41.690471    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2" event={"ID":"1db3bbf8-847c-4004-8b1b-91bca311dbf7","Type":"ContainerStarted","Data":"f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995"}
Oct 30 19:28:41.690943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:41.690533    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2" event={"ID":"1db3bbf8-847c-4004-8b1b-91bca311dbf7","Type":"ContainerStarted","Data":"a5d0ba2c48d35732573d6070be6f4cce7ea973e26d01d6ec098f0a7fb726fb96"}
Oct 30 19:28:41.690943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:41.690561    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:41.692905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:41.692870    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b" event={"ID":"eb1df337-9ecf-43f5-95ad-99faa700a4b0","Type":"ContainerStarted","Data":"40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda"}
Oct 30 19:28:41.693086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:41.692920    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b" event={"ID":"eb1df337-9ecf-43f5-95ad-99faa700a4b0","Type":"ContainerStarted","Data":"336a80e40d25a4c27e2bb11b34f5e8517616ee506e80d3ca0a48903e737a02de"}
Oct 30 19:28:41.693161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:41.693095    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:41.715262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:41.715204    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2" podStartSLOduration=1.71518533 podStartE2EDuration="1.71518533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:41.712273455 +0000 UTC m=+1303.709188840" watchObservedRunningTime="2026-10-30 19:28:41.71518533 +0000 UTC m=+1303.712100732"
Oct 30 19:28:41.733871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:41.733815    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b" podStartSLOduration=1.7337993310000002 podStartE2EDuration="1.733799331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:41.730955996 +0000 UTC m=+1303.727871382" watchObservedRunningTime="2026-10-30 19:28:41.733799331 +0000 UTC m=+1303.730714740"
Oct 30 19:28:43.647440 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:28:43.647397    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:28:47.203548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:47.203508    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:28:47.637117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:47.636324    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:28:47.637117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:47.636800    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:28:50.717009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.716973    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"]
Oct 30 19:28:50.724443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.724413    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.739457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.738956    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f61e62-openshift-default-dockercfg-r85hx\""
Oct 30 19:28:50.742450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.742422    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"]
Oct 30 19:28:50.841987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.841949    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phkvb\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-kube-api-access-phkvb\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.842206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.842017    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-envoy\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.842206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.842125    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-socket\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.842291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.842257    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-certs\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.842340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.842324    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e939f83-c3c3-424c-a26b-a20a197950ec-istiod-ca-cert\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.842423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.842410    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-token\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.842463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.842437    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-podinfo\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.842539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.842523    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-data\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.842612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.842596    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-credential-socket\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.957079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.956115    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-token\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.957079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.956180    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-podinfo\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.957079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.956281    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-data\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.957079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.956361    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-credential-socket\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.982147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.972124    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-data\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:50.982147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.978442    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-credential-socket\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.013078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:50.997693    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-podinfo\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.013078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.003753    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-token\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.013078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.004360    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkvb\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-kube-api-access-phkvb\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.013078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.004802    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-envoy\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.013078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.004839    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-socket\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.013078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.005003    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-certs\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.013078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.005125    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e939f83-c3c3-424c-a26b-a20a197950ec-istiod-ca-cert\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.013078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.009382    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-socket\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.013078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.009769    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-certs\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.020122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.014773    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e939f83-c3c3-424c-a26b-a20a197950ec-istiod-ca-cert\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.037079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.032672    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-envoy\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.072081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.062208    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phkvb\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-kube-api-access-phkvb\") pod \"e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") " pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.359172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.359138    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:51.683564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.683534    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"]
Oct 30 19:28:51.685298 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:28:51.685262    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e939f83_c3c3_424c_a26b_a20a197950ec.slice/crio-5eade939c24aaae8bc422c36b0550998072ec67eeff0501481309e23a3fd3f24 WatchSource:0}: Error finding container 5eade939c24aaae8bc422c36b0550998072ec67eeff0501481309e23a3fd3f24: Status 404 returned error can't find the container with id 5eade939c24aaae8bc422c36b0550998072ec67eeff0501481309e23a3fd3f24
Oct 30 19:28:51.690910 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.690869    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:51.691044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.690969    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:51.691044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.691021    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:28:51.877372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.877334    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" event={"ID":"4e939f83-c3c3-424c-a26b-a20a197950ec","Type":"ContainerStarted","Data":"cb4234e3e81b3ea5aa9bc06cc0e4878c43fca4f73c5589deb56bc90ed341d898"}
Oct 30 19:28:51.877765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.877399    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" event={"ID":"4e939f83-c3c3-424c-a26b-a20a197950ec","Type":"ContainerStarted","Data":"5eade939c24aaae8bc422c36b0550998072ec67eeff0501481309e23a3fd3f24"}
Oct 30 19:28:51.910977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:51.910903    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" podStartSLOduration=1.9108826890000001 podStartE2EDuration="1.910882689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:51.905190631 +0000 UTC m=+1313.902106017" watchObservedRunningTime="2026-10-30 19:28:51.910882689 +0000 UTC m=+1313.907798076"
Oct 30 19:28:52.360227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:52.360189    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:52.366523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:52.366499    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:52.705778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:52.705691    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:28:52.705778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:52.705772    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:28:52.886258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:52.886228    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:52.887288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:52.887265    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:28:57.039069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:57.039014    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"]
Oct 30 19:28:57.043752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:57.043718    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697c73c2-q6hsr"]
Oct 30 19:28:58.115878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:58.115846    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf931055-5000-4c9e-93ff-06c113435975" path="/var/lib/kubelet/pods/cf931055-5000-4c9e-93ff-06c113435975/volumes"
Oct 30 19:28:58.946622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:58.946587    6777 scope.go:122] "RemoveContainer" containerID="75e1418a9369c04aaab330785ce954e423bb01263cc0780375f44fd51efad5d9"
Oct 30 19:28:58.959628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:58.959605    6777 scope.go:122] "RemoveContainer" containerID="fbdaa248cecb111164ed799b3da4890a8925f2b654cd069e4a46707d3e1bdb3d"
Oct 30 19:28:58.970718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:58.970697    6777 scope.go:122] "RemoveContainer" containerID="309613fc5b47d0a01804fac6c9ec8150b1b37d863dd3cca7ce3c312779a2d5ad"
Oct 30 19:28:58.981170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:58.981144    6777 scope.go:122] "RemoveContainer" containerID="a0e0e3d121f35cc8d2b52d9c43fa37430c03a67cd13b29073d6ce337407145a9"
Oct 30 19:28:58.992219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:28:58.992200    6777 scope.go:122] "RemoveContainer" containerID="0c87708395f9fa838cf00d89f1bf392f172e038741bd8f3e11a58fc3ae70ee3f"
Oct 30 19:29:00.183183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.183147    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"]
Oct 30 19:29:00.183624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.183420    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy" containerID="cri-o://615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018" gracePeriod=30
Oct 30 19:29:00.683685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.683649    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"]
Oct 30 19:29:00.691306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.691274    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:00.696854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.696824    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"]
Oct 30 19:29:00.731791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.731762    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:00.731976 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.731824    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:29:00.735513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.735493    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:00.735656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.735642    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:00.735890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.735866    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvx6r\" (UniqueName: \"kubernetes.io/projected/5567eeb1-a32b-4f01-8895-e9bcb3685f51-kube-api-access-mvx6r\") pod \"maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:00.836947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.836900    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:00.837203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.837179    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvx6r\" (UniqueName: \"kubernetes.io/projected/5567eeb1-a32b-4f01-8895-e9bcb3685f51-kube-api-access-mvx6r\") pod \"maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:00.837349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.837334    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:00.838000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.837978    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:00.839392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.839374    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:00.850091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:00.850044    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvx6r\" (UniqueName: \"kubernetes.io/projected/5567eeb1-a32b-4f01-8895-e9bcb3685f51-kube-api-access-mvx6r\") pod \"maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:01.009129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:01.009029    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:01.314113 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:29:01.314078    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5567eeb1_a32b_4f01_8895_e9bcb3685f51.slice/crio-cf186d5597a746097e82e5d9e5fb53239fe065fecdc4d91865cb9ee349531f74 WatchSource:0}: Error finding container cf186d5597a746097e82e5d9e5fb53239fe065fecdc4d91865cb9ee349531f74: Status 404 returned error can't find the container with id cf186d5597a746097e82e5d9e5fb53239fe065fecdc4d91865cb9ee349531f74
Oct 30 19:29:01.314993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:01.314960    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"]
Oct 30 19:29:01.992018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:01.991916    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd" event={"ID":"5567eeb1-a32b-4f01-8895-e9bcb3685f51","Type":"ContainerStarted","Data":"24ab41d151ffbd81f1b2ed5d1d72778b115381994047ee64be10c82f79e9116b"}
Oct 30 19:29:01.992018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:01.992012    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd" event={"ID":"5567eeb1-a32b-4f01-8895-e9bcb3685f51","Type":"ContainerStarted","Data":"cf186d5597a746097e82e5d9e5fb53239fe065fecdc4d91865cb9ee349531f74"}
Oct 30 19:29:02.029882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:02.029826    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd" podStartSLOduration=2.029811155 podStartE2EDuration="2.029811155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:02.023434224 +0000 UTC m=+1324.020349612" watchObservedRunningTime="2026-10-30 19:29:02.029811155 +0000 UTC m=+1324.026726540"
Oct 30 19:29:03.022148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:03.017340    6777 generic.go:356] "Generic (PLEG): container finished" podID="5567eeb1-a32b-4f01-8895-e9bcb3685f51" containerID="24ab41d151ffbd81f1b2ed5d1d72778b115381994047ee64be10c82f79e9116b" exitCode=0
Oct 30 19:29:03.022148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:03.017587    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd" event={"ID":"5567eeb1-a32b-4f01-8895-e9bcb3685f51","Type":"ContainerDied","Data":"24ab41d151ffbd81f1b2ed5d1d72778b115381994047ee64be10c82f79e9116b"}
Oct 30 19:29:04.220527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.220500    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:04.277048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.276954    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-openshift-ca\") pod \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") "
Oct 30 19:29:04.277274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.277135    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5567eeb1-a32b-4f01-8895-e9bcb3685f51-kube-api-access-mvx6r\" (UniqueName: \"kubernetes.io/projected/5567eeb1-a32b-4f01-8895-e9bcb3685f51-kube-api-access-mvx6r\") pod \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") "
Oct 30 19:29:04.277274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.277193    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-xks-ca\") pod \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\" (UID: \"5567eeb1-a32b-4f01-8895-e9bcb3685f51\") "
Oct 30 19:29:04.277426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.277394    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-openshift-ca" pod "5567eeb1-a32b-4f01-8895-e9bcb3685f51" (UID: "5567eeb1-a32b-4f01-8895-e9bcb3685f51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:04.278189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.278163    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:04.279432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.279408    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5567eeb1-a32b-4f01-8895-e9bcb3685f51-kube-api-access-mvx6r" pod "5567eeb1-a32b-4f01-8895-e9bcb3685f51" (UID: "5567eeb1-a32b-4f01-8895-e9bcb3685f51"). InnerVolumeSpecName "kube-api-access-mvx6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:04.279782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.279768    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-xks-ca" pod "5567eeb1-a32b-4f01-8895-e9bcb3685f51" (UID: "5567eeb1-a32b-4f01-8895-e9bcb3685f51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:04.380707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.380668    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvx6r\" (UniqueName: \"kubernetes.io/projected/5567eeb1-a32b-4f01-8895-e9bcb3685f51-kube-api-access-mvx6r\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:04.380992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:04.380971    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567eeb1-a32b-4f01-8895-e9bcb3685f51-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:05.035972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.035941    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"
Oct 30 19:29:05.036157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.036017    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd" event={"ID":"5567eeb1-a32b-4f01-8895-e9bcb3685f51","Type":"ContainerDied","Data":"cf186d5597a746097e82e5d9e5fb53239fe065fecdc4d91865cb9ee349531f74"}
Oct 30 19:29:05.036157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.036090    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf186d5597a746097e82e5d9e5fb53239fe065fecdc4d91865cb9ee349531f74"
Oct 30 19:29:05.491353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.491325    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:29:05.614847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.614761    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-kube-api-access-9nvtd\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-kube-api-access-9nvtd\") pod \"cb33245b-7109-4488-8a23-9633a63c42dc\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") "
Oct 30 19:29:05.615072 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.614874    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-istio-token\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-istio-token\") pod \"cb33245b-7109-4488-8a23-9633a63c42dc\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") "
Oct 30 19:29:05.615072 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.615039    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb33245b-7109-4488-8a23-9633a63c42dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb33245b-7109-4488-8a23-9633a63c42dc-istiod-ca-cert\") pod \"cb33245b-7109-4488-8a23-9633a63c42dc\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") "
Oct 30 19:29:05.615206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.615168    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-socket\") pod \"cb33245b-7109-4488-8a23-9633a63c42dc\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") "
Oct 30 19:29:05.615263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.615230    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-data\") pod \"cb33245b-7109-4488-8a23-9633a63c42dc\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") "
Oct 30 19:29:05.615318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.615293    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-certs\") pod \"cb33245b-7109-4488-8a23-9633a63c42dc\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") "
Oct 30 19:29:05.615371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.615320    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-envoy\") pod \"cb33245b-7109-4488-8a23-9633a63c42dc\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") "
Oct 30 19:29:05.615492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.615474    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb33245b-7109-4488-8a23-9633a63c42dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb33245b-7109-4488-8a23-9633a63c42dc-istio-podinfo\") pod \"cb33245b-7109-4488-8a23-9633a63c42dc\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") "
Oct 30 19:29:05.615593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.615542    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-credential-socket\") pod \"cb33245b-7109-4488-8a23-9633a63c42dc\" (UID: \"cb33245b-7109-4488-8a23-9633a63c42dc\") "
Oct 30 19:29:05.616520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.616478    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-credential-socket" pod "cb33245b-7109-4488-8a23-9633a63c42dc" (UID: "cb33245b-7109-4488-8a23-9633a63c42dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:05.617088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.617038    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-data" pod "cb33245b-7109-4488-8a23-9633a63c42dc" (UID: "cb33245b-7109-4488-8a23-9633a63c42dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:05.617583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.617560    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-certs" pod "cb33245b-7109-4488-8a23-9633a63c42dc" (UID: "cb33245b-7109-4488-8a23-9633a63c42dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:05.617870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.617846    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb33245b-7109-4488-8a23-9633a63c42dc-istiod-ca-cert" pod "cb33245b-7109-4488-8a23-9633a63c42dc" (UID: "cb33245b-7109-4488-8a23-9633a63c42dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:05.620539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.620512    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-envoy" pod "cb33245b-7109-4488-8a23-9633a63c42dc" (UID: "cb33245b-7109-4488-8a23-9633a63c42dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:05.625836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.625803    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb33245b-7109-4488-8a23-9633a63c42dc-istio-podinfo" pod "cb33245b-7109-4488-8a23-9633a63c42dc" (UID: "cb33245b-7109-4488-8a23-9633a63c42dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:29:05.625985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.625947    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-kube-api-access-9nvtd" pod "cb33245b-7109-4488-8a23-9633a63c42dc" (UID: "cb33245b-7109-4488-8a23-9633a63c42dc"). InnerVolumeSpecName "kube-api-access-9nvtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:05.626639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.626598    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-istio-token" pod "cb33245b-7109-4488-8a23-9633a63c42dc" (UID: "cb33245b-7109-4488-8a23-9633a63c42dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:05.628337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.628313    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-socket" pod "cb33245b-7109-4488-8a23-9633a63c42dc" (UID: "cb33245b-7109-4488-8a23-9633a63c42dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:05.718700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.718661    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nvtd\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-kube-api-access-9nvtd\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:05.718700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.718694    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb33245b-7109-4488-8a23-9633a63c42dc-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:05.718700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.718703    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb33245b-7109-4488-8a23-9633a63c42dc-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:05.718988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.718712    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:05.718988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.718720    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:05.718988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.718730    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:05.718988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.718743    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:05.718988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.718756    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb33245b-7109-4488-8a23-9633a63c42dc-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:05.718988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:05.718765    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb33245b-7109-4488-8a23-9633a63c42dc-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:06.036246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.036214    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"]
Oct 30 19:29:06.040689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.040656    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a160f2d-49w89"]
Oct 30 19:29:06.046314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.046286    6777 generic.go:356] "Generic (PLEG): container finished" podID="cb33245b-7109-4488-8a23-9633a63c42dc" containerID="615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018" exitCode=0
Oct 30 19:29:06.046457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.046339    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"
Oct 30 19:29:06.046457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.046367    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" event={"ID":"cb33245b-7109-4488-8a23-9633a63c42dc","Type":"ContainerDied","Data":"615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018"}
Oct 30 19:29:06.046457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.046424    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k" event={"ID":"cb33245b-7109-4488-8a23-9633a63c42dc","Type":"ContainerDied","Data":"08f7831cc56ebd1ae72b29194966c44579a586961c6a24c44d0d2d119e323fb7"}
Oct 30 19:29:06.046592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.046456    6777 scope.go:122] "RemoveContainer" containerID="615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018"
Oct 30 19:29:06.071372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.071352    6777 scope.go:122] "RemoveContainer" containerID="00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e"
Oct 30 19:29:06.082843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.082813    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"]
Oct 30 19:29:06.087912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.087875    6777 scope.go:122] "RemoveContainer" containerID="615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018"
Oct 30 19:29:06.088487 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:06.088460    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018\": container with ID starting with 615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018 not found: ID does not exist" containerID="615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018"
Oct 30 19:29:06.088606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.088494    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018"} err="failed to get container status \"615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018\": rpc error: code = NotFound desc = could not find container \"615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018\": container with ID starting with 615ea9f66e75ec5d3d50eb038307ee085113292f4b7d1fd32a2927c6d2731018 not found: ID does not exist"
Oct 30 19:29:06.088606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.088512    6777 scope.go:122] "RemoveContainer" containerID="00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e"
Oct 30 19:29:06.088847 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:06.088830    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e\": container with ID starting with 00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e not found: ID does not exist" containerID="00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e"
Oct 30 19:29:06.088959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.088860    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e"} err="failed to get container status \"00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e\": rpc error: code = NotFound desc = could not find container \"00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e\": container with ID starting with 00e1c24c5f83fa124ff2cfe970e786a3f60eb5bfbb438f624f1038e01a81f21e not found: ID does not exist"
Oct 30 19:29:06.089001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.088979    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9a718a-openshift-default-698df85f8c-vf45k"]
Oct 30 19:29:06.114563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.114531    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549e3e83-1f92-4842-854e-0cf02b70294a" path="/var/lib/kubelet/pods/549e3e83-1f92-4842-854e-0cf02b70294a/volumes"
Oct 30 19:29:06.115142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:06.115128    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" path="/var/lib/kubelet/pods/cb33245b-7109-4488-8a23-9633a63c42dc/volumes"
Oct 30 19:29:07.921443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.921411    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"]
Oct 30 19:29:07.922710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.922668    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy"
Oct 30 19:29:07.922710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.922691    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy"
Oct 30 19:29:07.922710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.922716    6777 cpu_manager.go:414] "containerMap: removing container" podUID="5567eeb1-a32b-4f01-8895-e9bcb3685f51" containerName="revoke-keys"
Oct 30 19:29:07.923079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.922724    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5567eeb1-a32b-4f01-8895-e9bcb3685f51" containerName="revoke-keys"
Oct 30 19:29:07.923216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.923177    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="5567eeb1-a32b-4f01-8895-e9bcb3685f51" containerName="revoke-keys"
Oct 30 19:29:07.923299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.923217    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy"
Oct 30 19:29:07.923299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.923234    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy"
Oct 30 19:29:07.930206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.930178    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"]
Oct 30 19:29:07.930332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.930284    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:07.931347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.931324    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy"
Oct 30 19:29:07.931347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.931346    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb33245b-7109-4488-8a23-9633a63c42dc" containerName="istio-proxy"
Oct 30 19:29:07.938571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.938544    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"]
Oct 30 19:29:07.938731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.938660    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:07.943423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.943390    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"]
Oct 30 19:29:07.948850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.948822    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8e6159\""
Oct 30 19:29:07.948964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:07.948938    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8e6159-dockercfg-2f994\""
Oct 30 19:29:08.110581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.110540    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5128a10-1b65-4461-be9b-3903830e5dbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr\" (UID: \"a5128a10-1b65-4461-be9b-3903830e5dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:08.110906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.110873    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7tbc\" (UniqueName: \"kubernetes.io/projected/56180ed3-f18c-4df7-a682-7cf948382c57-kube-api-access-s7tbc\") pod \"payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf\" (UID: \"56180ed3-f18c-4df7-a682-7cf948382c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:08.111451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.111402    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54t6q\" (UniqueName: \"kubernetes.io/projected/a5128a10-1b65-4461-be9b-3903830e5dbe-kube-api-access-54t6q\") pod \"payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr\" (UID: \"a5128a10-1b65-4461-be9b-3903830e5dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:08.111678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.111645    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56180ed3-f18c-4df7-a682-7cf948382c57-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf\" (UID: \"56180ed3-f18c-4df7-a682-7cf948382c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:08.255656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.255563    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7tbc\" (UniqueName: \"kubernetes.io/projected/56180ed3-f18c-4df7-a682-7cf948382c57-kube-api-access-s7tbc\") pod \"payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf\" (UID: \"56180ed3-f18c-4df7-a682-7cf948382c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:08.265857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.265820    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54t6q\" (UniqueName: \"kubernetes.io/projected/a5128a10-1b65-4461-be9b-3903830e5dbe-kube-api-access-54t6q\") pod \"payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr\" (UID: \"a5128a10-1b65-4461-be9b-3903830e5dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:08.266132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.266113    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56180ed3-f18c-4df7-a682-7cf948382c57-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf\" (UID: \"56180ed3-f18c-4df7-a682-7cf948382c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:08.268709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.267313    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56180ed3-f18c-4df7-a682-7cf948382c57-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf\" (UID: \"56180ed3-f18c-4df7-a682-7cf948382c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:08.282086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.269308    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5128a10-1b65-4461-be9b-3903830e5dbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr\" (UID: \"a5128a10-1b65-4461-be9b-3903830e5dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:08.285542 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.283137    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5128a10-1b65-4461-be9b-3903830e5dbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr\" (UID: \"a5128a10-1b65-4461-be9b-3903830e5dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:08.324371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.318196    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54t6q\" (UniqueName: \"kubernetes.io/projected/a5128a10-1b65-4461-be9b-3903830e5dbe-kube-api-access-54t6q\") pod \"payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr\" (UID: \"a5128a10-1b65-4461-be9b-3903830e5dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:08.337407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.328842    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7tbc\" (UniqueName: \"kubernetes.io/projected/56180ed3-f18c-4df7-a682-7cf948382c57-kube-api-access-s7tbc\") pod \"payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf\" (UID: \"56180ed3-f18c-4df7-a682-7cf948382c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:08.537592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.537556    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"]
Oct 30 19:29:08.537906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.537859    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb" podUID="f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" containerName="maas-api" containerID="cri-o://f6d2213ffa33f88c11fdf70b7359fc5e0c590481b6eac43070b961599820ee53" gracePeriod=30
Oct 30 19:29:08.573330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.573298    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:08.581122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:08.576465    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:09.259187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.259151    6777 generic.go:356] "Generic (PLEG): container finished" podID="f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" containerID="f6d2213ffa33f88c11fdf70b7359fc5e0c590481b6eac43070b961599820ee53" exitCode=0
Oct 30 19:29:09.259634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.259243    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb" event={"ID":"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a","Type":"ContainerDied","Data":"f6d2213ffa33f88c11fdf70b7359fc5e0c590481b6eac43070b961599820ee53"}
Oct 30 19:29:09.289524 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:29:09.289491    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5128a10_1b65_4461_be9b_3903830e5dbe.slice/crio-8a82fbb95a34bb773772532f87dd657bf6f48a64399170ddd3399e6a4433eb7c WatchSource:0}: Error finding container 8a82fbb95a34bb773772532f87dd657bf6f48a64399170ddd3399e6a4433eb7c: Status 404 returned error can't find the container with id 8a82fbb95a34bb773772532f87dd657bf6f48a64399170ddd3399e6a4433eb7c
Oct 30 19:29:09.315765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.315726    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"]
Oct 30 19:29:09.334511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.334372    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"]
Oct 30 19:29:09.357905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.357791    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:29:09.464090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.463145    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-maas-api-tls\") pod \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") "
Oct 30 19:29:09.464090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.463306    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-openshift-service-ca\") pod \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") "
Oct 30 19:29:09.464090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.463366    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-kube-api-access-kb5kk\" (UniqueName: \"kubernetes.io/projected/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-kube-api-access-kb5kk\") pod \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") "
Oct 30 19:29:09.464090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.463415    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-metrics-certs\") pod \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\" (UID: \"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a\") "
Oct 30 19:29:09.465102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.464744    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-openshift-service-ca" pod "f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" (UID: "f5ee0a79-5166-462f-8dea-7b2cc83eaf4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:09.467207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.467176    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-maas-api-tls" pod "f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" (UID: "f5ee0a79-5166-462f-8dea-7b2cc83eaf4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:09.467329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.467296    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-metrics-certs" pod "f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" (UID: "f5ee0a79-5166-462f-8dea-7b2cc83eaf4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:09.467394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.467341    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-kube-api-access-kb5kk" pod "f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" (UID: "f5ee0a79-5166-462f-8dea-7b2cc83eaf4a"). InnerVolumeSpecName "kube-api-access-kb5kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:09.565706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.565660    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-maas-api-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:09.565882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.565724    6777 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-openshift-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:09.565882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.565735    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb5kk\" (UniqueName: \"kubernetes.io/projected/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-kube-api-access-kb5kk\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:09.565882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:09.565745    6777 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a-metrics-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:10.038526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.038475    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"]
Oct 30 19:29:10.043564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.043533    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71dd47f-v6x86"]
Oct 30 19:29:10.116203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.116164    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0ae996-8882-4ef1-8dc1-0d242347943d" path="/var/lib/kubelet/pods/4a0ae996-8882-4ef1-8dc1-0d242347943d/volumes"
Oct 30 19:29:10.268904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.268871    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"
Oct 30 19:29:10.269419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.268877    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb" event={"ID":"f5ee0a79-5166-462f-8dea-7b2cc83eaf4a","Type":"ContainerDied","Data":"a6f679a1a39c198a4cc9b866d2e71d1e8d75faa48a143c7ac174c4d56488b40f"}
Oct 30 19:29:10.269419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.269024    6777 scope.go:122] "RemoveContainer" containerID="f6d2213ffa33f88c11fdf70b7359fc5e0c590481b6eac43070b961599820ee53"
Oct 30 19:29:10.270973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.270937    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf" event={"ID":"56180ed3-f18c-4df7-a682-7cf948382c57","Type":"ContainerStarted","Data":"1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b"}
Oct 30 19:29:10.271137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.270998    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf" event={"ID":"56180ed3-f18c-4df7-a682-7cf948382c57","Type":"ContainerStarted","Data":"3d56e09f8f6a7bd97491e3c245bd1e48e09916d966301324c487e60ec864bfa0"}
Oct 30 19:29:10.271210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.271048    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:10.273028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.273009    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr" event={"ID":"a5128a10-1b65-4461-be9b-3903830e5dbe","Type":"ContainerStarted","Data":"bcf0830678465aafef3bc99f5acf6bbca86ad3130f65dad8b8b9c001588f20f1"}
Oct 30 19:29:10.273147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.273042    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr" event={"ID":"a5128a10-1b65-4461-be9b-3903830e5dbe","Type":"ContainerStarted","Data":"8a82fbb95a34bb773772532f87dd657bf6f48a64399170ddd3399e6a4433eb7c"}
Oct 30 19:29:10.273194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.273172    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:10.298862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.298817    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf" podStartSLOduration=3.298803154 podStartE2EDuration="3.298803154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:10.297906864 +0000 UTC m=+1332.294822250" watchObservedRunningTime="2026-10-30 19:29:10.298803154 +0000 UTC m=+1332.295718538"
Oct 30 19:29:10.318541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.318507    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"]
Oct 30 19:29:10.324456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.324430    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48b1a6a7-7bd465bcc9-lsgkb"]
Oct 30 19:29:10.342931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:10.342880    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr" podStartSLOduration=3.342866262 podStartE2EDuration="3.342866262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:10.339514667 +0000 UTC m=+1332.336430054" watchObservedRunningTime="2026-10-30 19:29:10.342866262 +0000 UTC m=+1332.339781646"
Oct 30 19:29:12.118421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:12.118380    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" path="/var/lib/kubelet/pods/f5ee0a79-5166-462f-8dea-7b2cc83eaf4a/volumes"
Oct 30 19:29:13.641657 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:13.641616    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:29:14.945014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:14.944982    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"]
Oct 30 19:29:14.947007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:14.946049    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" containerName="maas-api"
Oct 30 19:29:14.947007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:14.946102    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" containerName="maas-api"
Oct 30 19:29:14.947007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:14.946395    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ee0a79-5166-462f-8dea-7b2cc83eaf4a" containerName="maas-api"
Oct 30 19:29:14.955761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:14.955730    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:14.968702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:14.968665    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"]
Oct 30 19:29:14.972805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:14.972774    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f61e62-serving-cert\""
Oct 30 19:29:15.045856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.045817    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-metrics-certs\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.046031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.045864    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9rs\" (UniqueName: \"kubernetes.io/projected/384ea436-9ae8-493f-94ee-2af351c41196-kube-api-access-8n9rs\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.046031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.046004    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384ea436-9ae8-493f-94ee-2af351c41196-openshift-service-ca\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.046169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.046147    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.148711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.148667    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-metrics-certs\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.172085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.170133    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9rs\" (UniqueName: \"kubernetes.io/projected/384ea436-9ae8-493f-94ee-2af351c41196-kube-api-access-8n9rs\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.172085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.170284    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384ea436-9ae8-493f-94ee-2af351c41196-openshift-service-ca\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.172085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.170428    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.172085 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:15.170741    6777 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-serving-cert: secret "maas-api-e2e-api-b-f61e62-serving-cert" not found
Oct 30 19:29:15.172085 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:15.170825    6777 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls podName:384ea436-9ae8-493f-94ee-2af351c41196 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:15.670801846 +0000 UTC m=+1337.667717226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls") pod "maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg" (UID: "384ea436-9ae8-493f-94ee-2af351c41196") : secret "maas-api-e2e-api-b-f61e62-serving-cert" not found
Oct 30 19:29:15.173361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.173329    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384ea436-9ae8-493f-94ee-2af351c41196-openshift-service-ca\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.193722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.193650    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-metrics-certs\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.249138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.246025    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9rs\" (UniqueName: \"kubernetes.io/projected/384ea436-9ae8-493f-94ee-2af351c41196-kube-api-access-8n9rs\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.693634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:15.693600    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:15.693835 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:15.693756    6777 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-serving-cert: secret "maas-api-e2e-api-b-f61e62-serving-cert" not found
Oct 30 19:29:15.693883 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:15.693836    6777 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls podName:384ea436-9ae8-493f-94ee-2af351c41196 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:16.693821309 +0000 UTC m=+1338.690736685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls") pod "maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg" (UID: "384ea436-9ae8-493f-94ee-2af351c41196") : secret "maas-api-e2e-api-b-f61e62-serving-cert" not found
Oct 30 19:29:16.723691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:16.723584    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:16.727033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:16.727001    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls\") pod \"maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:16.797599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:16.797565    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:17.089465 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:29:17.089430    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384ea436_9ae8_493f_94ee_2af351c41196.slice/crio-b785d983c98d393912a4eb5507790c3ceb0a3b9a4729ec8a7953b363e8dbb795 WatchSource:0}: Error finding container b785d983c98d393912a4eb5507790c3ceb0a3b9a4729ec8a7953b363e8dbb795: Status 404 returned error can't find the container with id b785d983c98d393912a4eb5507790c3ceb0a3b9a4729ec8a7953b363e8dbb795
Oct 30 19:29:17.089999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:17.089975    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"]
Oct 30 19:29:17.377698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:17.377612    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg" event={"ID":"384ea436-9ae8-493f-94ee-2af351c41196","Type":"ContainerStarted","Data":"b785d983c98d393912a4eb5507790c3ceb0a3b9a4729ec8a7953b363e8dbb795"}
Oct 30 19:29:18.397798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:18.397759    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg" event={"ID":"384ea436-9ae8-493f-94ee-2af351c41196","Type":"ContainerStarted","Data":"a2be1eca4691db6ac9dd7a5700bb58c20f20a7b9bec732e7a37bfbd6a42ebf9b"}
Oct 30 19:29:18.398346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:18.397829    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:18.422624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:18.422569    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg" podStartSLOduration=3.9836978690000002 podStartE2EDuration="4.422553403s" totalImagesPullingTime="438.855534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:18.418858164 +0000 UTC m=+1340.415773550" watchObservedRunningTime="2026-10-30 19:29:18.422553403 +0000 UTC m=+1340.419468787"
Oct 30 19:29:21.030752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.030711    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"]
Oct 30 19:29:21.036258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.036228    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a43681-tq2xf"]
Oct 30 19:29:21.177615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.177574    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:29:21.219377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.219345    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"]
Oct 30 19:29:21.224629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.224600    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:21.232922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.232893    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-afdf16\""
Oct 30 19:29:21.233130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.232991    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-afdf16-dockercfg-9lt4p\""
Oct 30 19:29:21.241581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.241537    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"]
Oct 30 19:29:21.381139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.365005    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rps\" (UniqueName: \"kubernetes.io/projected/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-kube-api-access-f9rps\") pod \"payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg\" (UID: \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:21.409115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.387245    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:29:21.409115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.392471    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:29:21.409115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.405453    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg\" (UID: \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:21.565084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.518798    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rps\" (UniqueName: \"kubernetes.io/projected/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-kube-api-access-f9rps\") pod \"payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg\" (UID: \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:21.701883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.701789    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg\" (UID: \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:21.715140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.655294    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rps\" (UniqueName: \"kubernetes.io/projected/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-kube-api-access-f9rps\") pod \"payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg\" (UID: \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:21.715460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.703041    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg\" (UID: \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:21.864112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:21.864070    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:22.119271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:22.119231    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20de73d9-3deb-45dd-8294-ecbfc76bbd27" path="/var/lib/kubelet/pods/20de73d9-3deb-45dd-8294-ecbfc76bbd27/volumes"
Oct 30 19:29:22.187337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:22.185602    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"]
Oct 30 19:29:22.759091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:22.755169    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg" event={"ID":"950eeb12-a96d-4fac-b7c2-091cd8fa93c2","Type":"ContainerStarted","Data":"f29a1b6b4dbf891d9a41fdbe9666fc66ae4964d84de5d91a4a335a7147f216aa"}
Oct 30 19:29:22.759091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:22.755266    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg" event={"ID":"950eeb12-a96d-4fac-b7c2-091cd8fa93c2","Type":"ContainerStarted","Data":"41f72a751a2d7003b174be1bcfd89cd153c5d2fa2af9b8dc23da1df663f50ffc"}
Oct 30 19:29:22.759091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:22.756113    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:22.830134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:22.820582    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg" podStartSLOduration=1.8205537550000002 podStartE2EDuration="1.820553755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:22.794503305 +0000 UTC m=+1344.791418701" watchObservedRunningTime="2026-10-30 19:29:22.820553755 +0000 UTC m=+1344.817469169"
Oct 30 19:29:24.069725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:24.068508    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"]
Oct 30 19:29:24.073971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:24.073941    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c759790d-n9p5w"]
Oct 30 19:29:24.118816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:24.118763    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acbb908a-4d2a-4f12-b556-78d023e6789d" path="/var/lib/kubelet/pods/acbb908a-4d2a-4f12-b556-78d023e6789d/volumes"
Oct 30 19:29:24.444593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:24.433299    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:29:33.773378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:33.773341    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:29:35.029535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:35.029487    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"]
Oct 30 19:29:35.034845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:35.034816    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f2d70b7-9nd8k"]
Oct 30 19:29:36.037530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:36.037497    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"]
Oct 30 19:29:36.041718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:36.041691    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cc615-5sd97"]
Oct 30 19:29:36.115569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:36.115531    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4085ae2d-9fe7-4d3e-b5eb-7623baa7714e" path="/var/lib/kubelet/pods/4085ae2d-9fe7-4d3e-b5eb-7623baa7714e/volumes"
Oct 30 19:29:36.116117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:36.116101    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbba48af-dcfc-4f28-a338-c5e6b09efc21" path="/var/lib/kubelet/pods/bbba48af-dcfc-4f28-a338-c5e6b09efc21/volumes"
Oct 30 19:29:41.599436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.599355    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"]
Oct 30 19:29:41.605988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.605909    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.611822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.611791    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"]
Oct 30 19:29:41.643918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.643883    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:41.644131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.643952    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:29:41.650024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.649991    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.650157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.650039    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7kz\" (UniqueName: \"kubernetes.io/projected/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-kube-api-access-cs7kz\") pod \"maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.650272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.650251    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.752157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.752117    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.752564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.752540    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.752633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.752589    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7kz\" (UniqueName: \"kubernetes.io/projected/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-kube-api-access-cs7kz\") pod \"maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.754125 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.753856    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.766138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.766104    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.766343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.766322    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7kz\" (UniqueName: \"kubernetes.io/projected/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-kube-api-access-cs7kz\") pod \"maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:41.924403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:41.924313    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:42.221669 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:29:42.221636    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a0c8c2_a2f8_4d86_a09c_49b3e85d9b3a.slice/crio-30363728ab230ad31f1ff8fe73ec3a7895e89f090f8516ad9b744f1392a5c10d WatchSource:0}: Error finding container 30363728ab230ad31f1ff8fe73ec3a7895e89f090f8516ad9b744f1392a5c10d: Status 404 returned error can't find the container with id 30363728ab230ad31f1ff8fe73ec3a7895e89f090f8516ad9b744f1392a5c10d
Oct 30 19:29:42.223165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:42.223134    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"]
Oct 30 19:29:43.167992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:43.167938    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9" event={"ID":"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a","Type":"ContainerStarted","Data":"38b2fe974e2c3412c11259640d0382026cbc1e7bd665d0cddc777e2d7d38c370"}
Oct 30 19:29:43.168466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:43.168004    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9" event={"ID":"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a","Type":"ContainerStarted","Data":"30363728ab230ad31f1ff8fe73ec3a7895e89f090f8516ad9b744f1392a5c10d"}
Oct 30 19:29:43.642272 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:43.642236    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:29:44.236092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:44.235570    6777 generic.go:356] "Generic (PLEG): container finished" podID="40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a" containerID="38b2fe974e2c3412c11259640d0382026cbc1e7bd665d0cddc777e2d7d38c370" exitCode=0
Oct 30 19:29:44.236092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:44.235915    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9" event={"ID":"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a","Type":"ContainerDied","Data":"38b2fe974e2c3412c11259640d0382026cbc1e7bd665d0cddc777e2d7d38c370"}
Oct 30 19:29:45.215225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.215171    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"]
Oct 30 19:29:45.222185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.222154    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b2e27-bzvdd"]
Oct 30 19:29:45.676370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.676342    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:45.808946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.808906    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-openshift-ca\") pod \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") "
Oct 30 19:29:45.809209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.809191    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-xks-ca\") pod \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") "
Oct 30 19:29:45.809340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.809317    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-openshift-ca" pod "40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a" (UID: "40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:45.809483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.809462    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-kube-api-access-cs7kz\" (UniqueName: \"kubernetes.io/projected/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-kube-api-access-cs7kz\") pod \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\" (UID: \"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a\") "
Oct 30 19:29:45.810197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.810172    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:45.811431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.811403    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-xks-ca" pod "40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a" (UID: "40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:45.811521 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.811489    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-kube-api-access-cs7kz" pod "40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a" (UID: "40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a"). InnerVolumeSpecName "kube-api-access-cs7kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:45.912331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.912290    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:45.912331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:45.912333    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7kz\" (UniqueName: \"kubernetes.io/projected/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a-kube-api-access-cs7kz\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:46.220383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:46.218788    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5567eeb1-a32b-4f01-8895-e9bcb3685f51" path="/var/lib/kubelet/pods/5567eeb1-a32b-4f01-8895-e9bcb3685f51/volumes"
Oct 30 19:29:46.334693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:46.334638    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9" event={"ID":"40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a","Type":"ContainerDied","Data":"30363728ab230ad31f1ff8fe73ec3a7895e89f090f8516ad9b744f1392a5c10d"}
Oct 30 19:29:46.335290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:46.335268    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30363728ab230ad31f1ff8fe73ec3a7895e89f090f8516ad9b744f1392a5c10d"
Oct 30 19:29:46.335851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:46.335205    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"
Oct 30 19:29:48.047853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:48.047742    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"]
Oct 30 19:29:48.085332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:48.085285    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac91a75-bcmz5"]
Oct 30 19:29:48.117630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:48.117578    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93692622-047d-43d5-a55c-9fa4d5d718b6" path="/var/lib/kubelet/pods/93692622-047d-43d5-a55c-9fa4d5d718b6/volumes"
Oct 30 19:29:51.786427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:51.785799    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"]
Oct 30 19:29:53.935353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:53.935313    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"]
Oct 30 19:29:53.935879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:53.935593    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy" containerID="cri-o://1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c" gracePeriod=30
Oct 30 19:29:56.786331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:56.786263    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c" podUID="86652ae5-47e8-49f3-afd6-4c75416ce45b" containerName="payload-processing" containerID="cri-o://ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e" gracePeriod=25
Oct 30 19:29:57.137180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.137152    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:29:57.206920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.206881    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86652ae5-47e8-49f3-afd6-4c75416ce45b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86652ae5-47e8-49f3-afd6-4c75416ce45b-plugins-config-volume\") pod \"86652ae5-47e8-49f3-afd6-4c75416ce45b\" (UID: \"86652ae5-47e8-49f3-afd6-4c75416ce45b\") "
Oct 30 19:29:57.207482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.207449    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86652ae5-47e8-49f3-afd6-4c75416ce45b-kube-api-access-g6prl\" (UniqueName: \"kubernetes.io/projected/86652ae5-47e8-49f3-afd6-4c75416ce45b-kube-api-access-g6prl\") pod \"86652ae5-47e8-49f3-afd6-4c75416ce45b\" (UID: \"86652ae5-47e8-49f3-afd6-4c75416ce45b\") "
Oct 30 19:29:57.209372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.209327    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86652ae5-47e8-49f3-afd6-4c75416ce45b-plugins-config-volume" pod "86652ae5-47e8-49f3-afd6-4c75416ce45b" (UID: "86652ae5-47e8-49f3-afd6-4c75416ce45b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:57.212329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.212299    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86652ae5-47e8-49f3-afd6-4c75416ce45b-kube-api-access-g6prl" pod "86652ae5-47e8-49f3-afd6-4c75416ce45b" (UID: "86652ae5-47e8-49f3-afd6-4c75416ce45b"). InnerVolumeSpecName "kube-api-access-g6prl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:57.310695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.310644    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6prl\" (UniqueName: \"kubernetes.io/projected/86652ae5-47e8-49f3-afd6-4c75416ce45b-kube-api-access-g6prl\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:57.310695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.310687    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86652ae5-47e8-49f3-afd6-4c75416ce45b-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:57.581266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.581231    6777 generic.go:356] "Generic (PLEG): container finished" podID="86652ae5-47e8-49f3-afd6-4c75416ce45b" containerID="ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e" exitCode=0
Oct 30 19:29:57.581457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.581282    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"
Oct 30 19:29:57.581457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.581313    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c" event={"ID":"86652ae5-47e8-49f3-afd6-4c75416ce45b","Type":"ContainerDied","Data":"ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e"}
Oct 30 19:29:57.581457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.581373    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c" event={"ID":"86652ae5-47e8-49f3-afd6-4c75416ce45b","Type":"ContainerDied","Data":"a79b78eea267bbaaa5f6d7f1b46ea5f07e28e4f7e2c92b2d7bd96e7d783abead"}
Oct 30 19:29:57.581457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.581404    6777 scope.go:122] "RemoveContainer" containerID="ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e"
Oct 30 19:29:57.594716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.594692    6777 scope.go:122] "RemoveContainer" containerID="ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e"
Oct 30 19:29:57.595053 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:57.595031    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e\": container with ID starting with ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e not found: ID does not exist" containerID="ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e"
Oct 30 19:29:57.595168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.595074    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e"} err="failed to get container status \"ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e\": rpc error: code = NotFound desc = could not find container \"ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e\": container with ID starting with ef9a69d1b118f64a5dad43e0ba8c4e79fa3f2bb596f2bb887a570d7d68243c3e not found: ID does not exist"
Oct 30 19:29:57.611334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.610189    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"]
Oct 30 19:29:57.615954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:57.615926    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ceda0-6ff4cc8588-xbn7c"]
Oct 30 19:29:58.117831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:58.117753    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86652ae5-47e8-49f3-afd6-4c75416ce45b" path="/var/lib/kubelet/pods/86652ae5-47e8-49f3-afd6-4c75416ce45b/volumes"
Oct 30 19:29:58.591903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:58.591873    6777 generic.go:356] "Generic (PLEG): container finished" podID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerID="cb4234e3e81b3ea5aa9bc06cc0e4878c43fca4f73c5589deb56bc90ed341d898" exitCode=0
Oct 30 19:29:58.592121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:58.591946    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" event={"ID":"4e939f83-c3c3-424c-a26b-a20a197950ec","Type":"ContainerDied","Data":"cb4234e3e81b3ea5aa9bc06cc0e4878c43fca4f73c5589deb56bc90ed341d898"}
Oct 30 19:29:58.592655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:58.592630    6777 scope.go:122] "RemoveContainer" containerID="cb4234e3e81b3ea5aa9bc06cc0e4878c43fca4f73c5589deb56bc90ed341d898"
Oct 30 19:29:58.616400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:58.597929    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:29:58.616400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:58.598030    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:29:58.616400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:58.598094    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:29:59.062050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.062016    6777 scope.go:122] "RemoveContainer" containerID="194a07df123bc668d6cb60f478da78934e1c2cbdf82e5b67f7b4ebe7e3be9729"
Oct 30 19:29:59.076556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.076515    6777 scope.go:122] "RemoveContainer" containerID="cf2de65edaf38017860f79eb9bce6cbf6ac94fe02e60e333c7d61a7f9aae1a81"
Oct 30 19:29:59.097787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.097637    6777 scope.go:122] "RemoveContainer" containerID="676b05b1790902d8b72140276c9cfda229c1532f82451e08fccbce0951976062"
Oct 30 19:29:59.115532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.115419    6777 scope.go:122] "RemoveContainer" containerID="c2de734de49df6925f63e14963329d8091b7e3abb2d361f2c927a5c2cd815cc7"
Oct 30 19:29:59.307617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.307591    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:29:59.443679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.443645    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"]
Oct 30 19:29:59.444577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444556    6777 cpu_manager.go:414] "containerMap: removing container" podUID="40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a" containerName="revoke-keys"
Oct 30 19:29:59.444577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444577    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a" containerName="revoke-keys"
Oct 30 19:29:59.444779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444613    6777 cpu_manager.go:414] "containerMap: removing container" podUID="86652ae5-47e8-49f3-afd6-4c75416ce45b" containerName="payload-processing"
Oct 30 19:29:59.444779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444621    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86652ae5-47e8-49f3-afd6-4c75416ce45b" containerName="payload-processing"
Oct 30 19:29:59.444779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444653    6777 cpu_manager.go:414] "containerMap: removing container" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy"
Oct 30 19:29:59.444779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444661    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy"
Oct 30 19:29:59.444779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444697    6777 cpu_manager.go:414] "containerMap: removing container" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy"
Oct 30 19:29:59.444779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444704    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy"
Oct 30 19:29:59.445222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444894    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="86652ae5-47e8-49f3-afd6-4c75416ce45b" containerName="payload-processing"
Oct 30 19:29:59.445222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444912    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy"
Oct 30 19:29:59.445222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444928    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerName="istio-proxy"
Oct 30 19:29:59.445222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.444947    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a" containerName="revoke-keys"
Oct 30 19:29:59.511400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.511301    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"]
Oct 30 19:29:59.511582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.511443    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.519801    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-data\") pod \"81cb5dd9-9980-4623-9d2d-67f167639c95\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") "
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.519955    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-podinfo\") pod \"81cb5dd9-9980-4623-9d2d-67f167639c95\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") "
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.520083    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-socket\") pod \"81cb5dd9-9980-4623-9d2d-67f167639c95\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") "
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.520150    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-credential-socket\") pod \"81cb5dd9-9980-4623-9d2d-67f167639c95\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") "
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.520239    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-certs\") pod \"81cb5dd9-9980-4623-9d2d-67f167639c95\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") "
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.520414    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cb5dd9-9980-4623-9d2d-67f167639c95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cb5dd9-9980-4623-9d2d-67f167639c95-istiod-ca-cert\") pod \"81cb5dd9-9980-4623-9d2d-67f167639c95\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") "
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.520458    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-kube-api-access-j9st5\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-kube-api-access-j9st5\") pod \"81cb5dd9-9980-4623-9d2d-67f167639c95\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") "
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.520547    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-envoy\") pod \"81cb5dd9-9980-4623-9d2d-67f167639c95\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") "
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.520615    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-token\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-token\") pod \"81cb5dd9-9980-4623-9d2d-67f167639c95\" (UID: \"81cb5dd9-9980-4623-9d2d-67f167639c95\") "
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.520651    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-data" pod "81cb5dd9-9980-4623-9d2d-67f167639c95" (UID: "81cb5dd9-9980-4623-9d2d-67f167639c95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.520979    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-certs" pod "81cb5dd9-9980-4623-9d2d-67f167639c95" (UID: "81cb5dd9-9980-4623-9d2d-67f167639c95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521096    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24xgt\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-kube-api-access-24xgt\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521170    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-socket" pod "81cb5dd9-9980-4623-9d2d-67f167639c95" (UID: "81cb5dd9-9980-4623-9d2d-67f167639c95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521292    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521435    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521491    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521552    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521580    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521866    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521951    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.521988    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.522160    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.522179    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.522196    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.522441    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-credential-socket" pod "81cb5dd9-9980-4623-9d2d-67f167639c95" (UID: "81cb5dd9-9980-4623-9d2d-67f167639c95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.522713    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cb5dd9-9980-4623-9d2d-67f167639c95-istiod-ca-cert" pod "81cb5dd9-9980-4623-9d2d-67f167639c95" (UID: "81cb5dd9-9980-4623-9d2d-67f167639c95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.523383    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-02feaa-openshift-default-dockercfg-8qgr7\""
Oct 30 19:29:59.525351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.523812    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-kube-api-access-j9st5" pod "81cb5dd9-9980-4623-9d2d-67f167639c95" (UID: "81cb5dd9-9980-4623-9d2d-67f167639c95"). InnerVolumeSpecName "kube-api-access-j9st5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:59.526914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.526885    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-token" pod "81cb5dd9-9980-4623-9d2d-67f167639c95" (UID: "81cb5dd9-9980-4623-9d2d-67f167639c95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:59.529244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.529217    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-podinfo" pod "81cb5dd9-9980-4623-9d2d-67f167639c95" (UID: "81cb5dd9-9980-4623-9d2d-67f167639c95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:29:59.529372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.529318    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-envoy" pod "81cb5dd9-9980-4623-9d2d-67f167639c95" (UID: "81cb5dd9-9980-4623-9d2d-67f167639c95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:59.623317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623278    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.623498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623365    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.623498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623393    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.623498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623451    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xgt\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-kube-api-access-24xgt\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.623680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623559    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.623737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623704    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.623790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623752    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.623853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623814    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.623853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623846    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.623960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.623872    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.624162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624140    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.624293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624272    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:59.624365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624299    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:59.624365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624317    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81cb5dd9-9980-4623-9d2d-67f167639c95-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:59.624365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624332    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9st5\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-kube-api-access-j9st5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:59.624365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624348    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:59.624365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624362    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81cb5dd9-9980-4623-9d2d-67f167639c95-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:59.624642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624612    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.624881 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624861    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.625003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.624951    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.626197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.626176    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.626597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.626578    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.636264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.636240    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.636399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.636381    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xgt\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-kube-api-access-24xgt\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:29:59.685107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.685047    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" event={"ID":"4e939f83-c3c3-424c-a26b-a20a197950ec","Type":"ContainerStarted","Data":"63c138333167032010fd1a986dfc68804eeb5e583b544fff8395abfcd3a62d30"}
Oct 30 19:29:59.687150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.687098    6777 generic.go:356] "Generic (PLEG): container finished" podID="81cb5dd9-9980-4623-9d2d-67f167639c95" containerID="1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c" exitCode=0
Oct 30 19:29:59.687150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.687128    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"
Oct 30 19:29:59.687319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.687173    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" event={"ID":"81cb5dd9-9980-4623-9d2d-67f167639c95","Type":"ContainerDied","Data":"1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c"}
Oct 30 19:29:59.687319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.687223    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf" event={"ID":"81cb5dd9-9980-4623-9d2d-67f167639c95","Type":"ContainerDied","Data":"03974c5700b6e1b045acd2f6c190d6a7001461cdc89bca8c5789ec9f273a911d"}
Oct 30 19:29:59.687319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.687247    6777 scope.go:122] "RemoveContainer" containerID="1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c"
Oct 30 19:29:59.700051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.700030    6777 scope.go:122] "RemoveContainer" containerID="91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec"
Oct 30 19:29:59.712179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.712155    6777 scope.go:122] "RemoveContainer" containerID="1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c"
Oct 30 19:29:59.712695 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:59.712673    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c\": container with ID starting with 1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c not found: ID does not exist" containerID="1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c"
Oct 30 19:29:59.712805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.712708    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c"} err="failed to get container status \"1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c\": rpc error: code = NotFound desc = could not find container \"1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c\": container with ID starting with 1deb8c1635f1f58cb1f03b3266c35fb422ae513a62f7274f5bdc68dbc9a5739c not found: ID does not exist"
Oct 30 19:29:59.712805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.712734    6777 scope.go:122] "RemoveContainer" containerID="91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec"
Oct 30 19:29:59.713044 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:29:59.713024    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec\": container with ID starting with 91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec not found: ID does not exist" containerID="91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec"
Oct 30 19:29:59.713402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.713051    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec"} err="failed to get container status \"91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec\": rpc error: code = NotFound desc = could not find container \"91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec\": container with ID starting with 91530777324cf8133cab22696b7ab65d92546cccb38eaab1f2c6bed7352459ec not found: ID does not exist"
Oct 30 19:29:59.732159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.732128    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"]
Oct 30 19:29:59.735408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.735388    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9b2e27-openshift-default-8489bb687c-bcfjf"]
Oct 30 19:29:59.896763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:29:59.896730    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:30:00.371080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.363707    6777 patch_prober.go:29] interesting pod/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:00.371080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.363824    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 19:30:00.632935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.632731    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:30:00.632935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.632860    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:30:00.632935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.632904    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:30:00.639839 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.639806    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cb5dd9-9980-4623-9d2d-67f167639c95" path="/var/lib/kubelet/pods/81cb5dd9-9980-4623-9d2d-67f167639c95/volumes"
Oct 30 19:30:00.643191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.643116    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:30:00.643191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.643174    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"]
Oct 30 19:30:00.660596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.660560    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"]
Oct 30 19:30:00.666460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.665179    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:00.687234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.686306    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:30:00.724812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.724783    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"]
Oct 30 19:30:00.725218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.725196    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:00.734194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.734169    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"]
Oct 30 19:30:00.734392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.734265    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:00.735512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.734382    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" event={"ID":"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799","Type":"ContainerStarted","Data":"acc2a0c8d9db1842d645cc0fcc934b8215f6082f9a605f3ad38a24a5b5bbf84d"}
Oct 30 19:30:00.735647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.735635    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"]
Oct 30 19:30:00.735731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.735724    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"]
Oct 30 19:30:00.741688 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.741669    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"]
Oct 30 19:30:00.741892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.741780    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:00.748355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.748335    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"]
Oct 30 19:30:00.752350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.748437    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:00.754267 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.754234    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"]
Oct 30 19:30:00.754446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.754433    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"]
Oct 30 19:30:00.754591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.754578    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"]
Oct 30 19:30:00.754831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.754817    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:00.779142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.779103    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"]
Oct 30 19:30:00.783982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.783956    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"]
Oct 30 19:30:00.835308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.835272    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj\" (UID: \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:00.835565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.835357    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpxt5\" (UniqueName: \"kubernetes.io/projected/1276cd7c-e245-4663-a46f-d03854c463c7-kube-api-access-bpxt5\") pod \"maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s\" (UID: \"1276cd7c-e245-4663-a46f-d03854c463c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:00.835565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.835428    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8vf\" (UniqueName: \"kubernetes.io/projected/be08e6ae-ea90-4735-b444-211571ace69f-kube-api-access-6d8vf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd\" (UID: \"be08e6ae-ea90-4735-b444-211571ace69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:00.835565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.835512    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be08e6ae-ea90-4735-b444-211571ace69f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd\" (UID: \"be08e6ae-ea90-4735-b444-211571ace69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:00.835834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.835571    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2rd\" (UniqueName: \"kubernetes.io/projected/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-kube-api-access-fs2rd\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj\" (UID: \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:00.839242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.836591    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276cd7c-e245-4663-a46f-d03854c463c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s\" (UID: \"1276cd7c-e245-4663-a46f-d03854c463c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:00.943444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.943407    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276cd7c-e245-4663-a46f-d03854c463c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s\" (UID: \"1276cd7c-e245-4663-a46f-d03854c463c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:00.943757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.943739    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq\" (UID: \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:00.943883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.943870    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggbhz\" (UniqueName: \"kubernetes.io/projected/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-kube-api-access-ggbhz\") pod \"maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq\" (UID: \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:00.944191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.944162    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276cd7c-e245-4663-a46f-d03854c463c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s\" (UID: \"1276cd7c-e245-4663-a46f-d03854c463c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:00.944370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.944349    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtmhm\" (UniqueName: \"kubernetes.io/projected/173b9bfc-03f9-4604-b5b5-3f2d0625d231-kube-api-access-jtmhm\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4\" (UID: \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:00.944509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.944496    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnf6h\" (UniqueName: \"kubernetes.io/projected/569c8ef0-8451-4faa-895f-f4f497143b94-kube-api-access-vnf6h\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc\" (UID: \"569c8ef0-8451-4faa-895f-f4f497143b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:00.944671 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.944654    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj\" (UID: \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:00.944775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.944763    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/173b9bfc-03f9-4604-b5b5-3f2d0625d231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4\" (UID: \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:00.944880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.944868    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8ef0-8451-4faa-895f-f4f497143b94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc\" (UID: \"569c8ef0-8451-4faa-895f-f4f497143b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:00.944978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.944964    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpxt5\" (UniqueName: \"kubernetes.io/projected/1276cd7c-e245-4663-a46f-d03854c463c7-kube-api-access-bpxt5\") pod \"maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s\" (UID: \"1276cd7c-e245-4663-a46f-d03854c463c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:00.945137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.945120    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8vf\" (UniqueName: \"kubernetes.io/projected/be08e6ae-ea90-4735-b444-211571ace69f-kube-api-access-6d8vf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd\" (UID: \"be08e6ae-ea90-4735-b444-211571ace69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:00.945352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.945331    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj\" (UID: \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:00.945447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.945345    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be08e6ae-ea90-4735-b444-211571ace69f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd\" (UID: \"be08e6ae-ea90-4735-b444-211571ace69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:00.945553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.945540    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2rd\" (UniqueName: \"kubernetes.io/projected/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-kube-api-access-fs2rd\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj\" (UID: \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:00.946505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.946483    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be08e6ae-ea90-4735-b444-211571ace69f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd\" (UID: \"be08e6ae-ea90-4735-b444-211571ace69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:00.986791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.986756    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2rd\" (UniqueName: \"kubernetes.io/projected/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-kube-api-access-fs2rd\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj\" (UID: \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:00.988428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.988309    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8vf\" (UniqueName: \"kubernetes.io/projected/be08e6ae-ea90-4735-b444-211571ace69f-kube-api-access-6d8vf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd\" (UID: \"be08e6ae-ea90-4735-b444-211571ace69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:00.989206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:00.989170    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpxt5\" (UniqueName: \"kubernetes.io/projected/1276cd7c-e245-4663-a46f-d03854c463c7-kube-api-access-bpxt5\") pod \"maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s\" (UID: \"1276cd7c-e245-4663-a46f-d03854c463c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:01.026267 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.026238    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:01.047754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.047703    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtmhm\" (UniqueName: \"kubernetes.io/projected/173b9bfc-03f9-4604-b5b5-3f2d0625d231-kube-api-access-jtmhm\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4\" (UID: \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:01.048453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.048427    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnf6h\" (UniqueName: \"kubernetes.io/projected/569c8ef0-8451-4faa-895f-f4f497143b94-kube-api-access-vnf6h\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc\" (UID: \"569c8ef0-8451-4faa-895f-f4f497143b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:01.063015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.062980    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/173b9bfc-03f9-4604-b5b5-3f2d0625d231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4\" (UID: \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:01.064045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.064021    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8ef0-8451-4faa-895f-f4f497143b94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc\" (UID: \"569c8ef0-8451-4faa-895f-f4f497143b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:01.065890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.064821    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8ef0-8451-4faa-895f-f4f497143b94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc\" (UID: \"569c8ef0-8451-4faa-895f-f4f497143b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:01.065890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.063947    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/173b9bfc-03f9-4604-b5b5-3f2d0625d231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4\" (UID: \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:01.065890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.065187    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq\" (UID: \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:01.065890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.065285    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggbhz\" (UniqueName: \"kubernetes.io/projected/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-kube-api-access-ggbhz\") pod \"maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq\" (UID: \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:01.066233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.065901    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq\" (UID: \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:01.082909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.082874    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtmhm\" (UniqueName: \"kubernetes.io/projected/173b9bfc-03f9-4604-b5b5-3f2d0625d231-kube-api-access-jtmhm\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4\" (UID: \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:01.085925 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.085901    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnf6h\" (UniqueName: \"kubernetes.io/projected/569c8ef0-8451-4faa-895f-f4f497143b94-kube-api-access-vnf6h\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc\" (UID: \"569c8ef0-8451-4faa-895f-f4f497143b94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:01.086189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.086164    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggbhz\" (UniqueName: \"kubernetes.io/projected/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-kube-api-access-ggbhz\") pod \"maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq\" (UID: \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:01.120191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.120153    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:01.195620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.195306    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:01.208502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.208464    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:01.238428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.237629    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:01.240126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.240102    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:01.379083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.371178    6777 patch_prober.go:29] interesting pod/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:01.379083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.371244    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 19:30:01.385673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.385022    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"]
Oct 30 19:30:01.395134 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:30:01.391770    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe08e6ae_ea90_4735_b444_211571ace69f.slice/crio-7d9b883ef63c0f1bdb2085a8d970b9fb329584ec054d67df5eaac903876e2366 WatchSource:0}: Error finding container 7d9b883ef63c0f1bdb2085a8d970b9fb329584ec054d67df5eaac903876e2366: Status 404 returned error can't find the container with id 7d9b883ef63c0f1bdb2085a8d970b9fb329584ec054d67df5eaac903876e2366
Oct 30 19:30:01.630641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.630607    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"]
Oct 30 19:30:01.631169 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:30:01.631138    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173b9bfc_03f9_4604_b5b5_3f2d0625d231.slice/crio-55598d931309d35f18cb7d781e5511548d1e28bb395012cd4f1ff4e066ea2320 WatchSource:0}: Error finding container 55598d931309d35f18cb7d781e5511548d1e28bb395012cd4f1ff4e066ea2320: Status 404 returned error can't find the container with id 55598d931309d35f18cb7d781e5511548d1e28bb395012cd4f1ff4e066ea2320
Oct 30 19:30:01.688195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.688160    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"]
Oct 30 19:30:01.779657 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:30:01.779606    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569c8ef0_8451_4faa_895f_f4f497143b94.slice/crio-59aafb22a0a9c2c169c0b291e345bfcc5c01fbc19b9e052395fefc6cec290495 WatchSource:0}: Error finding container 59aafb22a0a9c2c169c0b291e345bfcc5c01fbc19b9e052395fefc6cec290495: Status 404 returned error can't find the container with id 59aafb22a0a9c2c169c0b291e345bfcc5c01fbc19b9e052395fefc6cec290495
Oct 30 19:30:01.875629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:01.875437    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"]
Oct 30 19:30:02.018204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.018111    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"]
Oct 30 19:30:02.385079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.377650    6777 patch_prober.go:29] interesting pod/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:02.385079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.377716    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 19:30:02.416670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.416632    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" event={"ID":"173b9bfc-03f9-4604-b5b5-3f2d0625d231","Type":"ContainerStarted","Data":"55598d931309d35f18cb7d781e5511548d1e28bb395012cd4f1ff4e066ea2320"}
Oct 30 19:30:02.416913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.416895    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"]
Oct 30 19:30:02.417037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.417026    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"]
Oct 30 19:30:02.417151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.417137    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" event={"ID":"1276cd7c-e245-4663-a46f-d03854c463c7","Type":"ContainerStarted","Data":"723d60594723e68af8e0a75967cb2ba62c7eb0086b2fab5f8c60dbed861da703"}
Oct 30 19:30:02.417268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.417255    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" event={"ID":"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799","Type":"ContainerStarted","Data":"14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3"}
Oct 30 19:30:02.419463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.419336    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" event={"ID":"be08e6ae-ea90-4735-b444-211571ace69f","Type":"ContainerStarted","Data":"7d9b883ef63c0f1bdb2085a8d970b9fb329584ec054d67df5eaac903876e2366"}
Oct 30 19:30:02.419887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.419835    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee94ead-brtdt"]
Oct 30 19:30:02.433800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.433767    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"]
Oct 30 19:30:02.441208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.441180    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy" containerID="cri-o://e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847" gracePeriod=30
Oct 30 19:30:02.501084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.495711    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podStartSLOduration=7.495686774 podStartE2EDuration="7.495686774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.47859832 +0000 UTC m=+1384.475513718" watchObservedRunningTime="2026-10-30 19:30:02.495686774 +0000 UTC m=+1384.492602161"
Oct 30 19:30:02.905180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.905112    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:30:02.953489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:02.953454    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:30:03.191868 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:03.191819    6777 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe08e6ae_ea90_4735_b444_211571ace69f.slice/crio-conmon-965e7160a51840fc7a972bd796c471d853e7fb487c8baa4d9a722cb2bd70b7ea.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173b9bfc_03f9_4604_b5b5_3f2d0625d231.slice/crio-conmon-3bea8cd9e157c5f13f8f0f709ecad6219bc3c119ba770c61d0f3209b167e67f1.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:30:03.381443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.381409    6777 patch_prober.go:29] interesting pod/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:03.381703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.381676    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 19:30:03.435787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.435686    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" event={"ID":"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c","Type":"ContainerStarted","Data":"53b0322b935f984419ccb7d52b2fe468a7e3b64185247560611b899896bb01e2"}
Oct 30 19:30:03.437599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.437572    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" event={"ID":"569c8ef0-8451-4faa-895f-f4f497143b94","Type":"ContainerStarted","Data":"59aafb22a0a9c2c169c0b291e345bfcc5c01fbc19b9e052395fefc6cec290495"}
Oct 30 19:30:03.439683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.439654    6777 generic.go:356] "Generic (PLEG): container finished" podID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerID="3bea8cd9e157c5f13f8f0f709ecad6219bc3c119ba770c61d0f3209b167e67f1" exitCode=60
Oct 30 19:30:03.439832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.439753    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" event={"ID":"173b9bfc-03f9-4604-b5b5-3f2d0625d231","Type":"ContainerDied","Data":"3bea8cd9e157c5f13f8f0f709ecad6219bc3c119ba770c61d0f3209b167e67f1"}
Oct 30 19:30:03.440253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.440219    6777 scope.go:122] "RemoveContainer" containerID="3bea8cd9e157c5f13f8f0f709ecad6219bc3c119ba770c61d0f3209b167e67f1"
Oct 30 19:30:03.442502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.442253    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" event={"ID":"1276cd7c-e245-4663-a46f-d03854c463c7","Type":"ContainerStarted","Data":"4ae2ad9e287517d89a3b28b96de9747b5e09e8c941241a1b9ed2dd25bf358b29"}
Oct 30 19:30:03.444032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.444007    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" event={"ID":"31fa5b2a-01b1-4cba-b726-cd9332da7b4a","Type":"ContainerStarted","Data":"79ea0414a900cd68cdc0676435b7e02817914407693584234a5dbe3adb24fec0"}
Oct 30 19:30:03.445961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.445941    6777 generic.go:356] "Generic (PLEG): container finished" podID="be08e6ae-ea90-4735-b444-211571ace69f" containerID="965e7160a51840fc7a972bd796c471d853e7fb487c8baa4d9a722cb2bd70b7ea" exitCode=60
Oct 30 19:30:03.446084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.446022    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" event={"ID":"be08e6ae-ea90-4735-b444-211571ace69f","Type":"ContainerDied","Data":"965e7160a51840fc7a972bd796c471d853e7fb487c8baa4d9a722cb2bd70b7ea"}
Oct 30 19:30:03.446292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.446277    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:30:03.446370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.446334    6777 scope.go:122] "RemoveContainer" containerID="965e7160a51840fc7a972bd796c471d853e7fb487c8baa4d9a722cb2bd70b7ea"
Oct 30 19:30:03.447334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.447317    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:30:03.529852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:03.529803    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" podStartSLOduration=3.5297893460000003 podStartE2EDuration="3.529789346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.527687674 +0000 UTC m=+1385.524603057" watchObservedRunningTime="2026-10-30 19:30:03.529789346 +0000 UTC m=+1385.526704730"
Oct 30 19:30:04.117610 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.117572    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0417cf65-b16e-46c2-b2c5-396fee491051" path="/var/lib/kubelet/pods/0417cf65-b16e-46c2-b2c5-396fee491051/volumes"
Oct 30 19:30:04.363505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.362489    6777 patch_prober.go:29] interesting pod/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:04.363505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.362550    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 19:30:04.491554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.491517    6777 generic.go:356] "Generic (PLEG): container finished" podID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerID="038493e75bafdeb3952ed81bf5dfd6fa65cf6e4bb56a3dad3bebaa6c5c4b6d5c" exitCode=60
Oct 30 19:30:04.491958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.491600    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" event={"ID":"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c","Type":"ContainerDied","Data":"038493e75bafdeb3952ed81bf5dfd6fa65cf6e4bb56a3dad3bebaa6c5c4b6d5c"}
Oct 30 19:30:04.492015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.491999    6777 scope.go:122] "RemoveContainer" containerID="038493e75bafdeb3952ed81bf5dfd6fa65cf6e4bb56a3dad3bebaa6c5c4b6d5c"
Oct 30 19:30:04.493665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.493637    6777 generic.go:356] "Generic (PLEG): container finished" podID="569c8ef0-8451-4faa-895f-f4f497143b94" containerID="91e31016f05890a7cc03bb2bce4e74911c30d53bdf43670400803924b65cb1cd" exitCode=60
Oct 30 19:30:04.494024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.493994    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" event={"ID":"569c8ef0-8451-4faa-895f-f4f497143b94","Type":"ContainerDied","Data":"91e31016f05890a7cc03bb2bce4e74911c30d53bdf43670400803924b65cb1cd"}
Oct 30 19:30:04.494331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.494314    6777 scope.go:122] "RemoveContainer" containerID="91e31016f05890a7cc03bb2bce4e74911c30d53bdf43670400803924b65cb1cd"
Oct 30 19:30:04.499277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.499206    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" event={"ID":"173b9bfc-03f9-4604-b5b5-3f2d0625d231","Type":"ContainerStarted","Data":"8aaf001d9e93f8f0240fcb2d8ad1cbc708abab2c78de39471cda7c0a0481efe4"}
Oct 30 19:30:04.501438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.501414    6777 generic.go:356] "Generic (PLEG): container finished" podID="1276cd7c-e245-4663-a46f-d03854c463c7" containerID="4ae2ad9e287517d89a3b28b96de9747b5e09e8c941241a1b9ed2dd25bf358b29" exitCode=60
Oct 30 19:30:04.501557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.501506    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" event={"ID":"1276cd7c-e245-4663-a46f-d03854c463c7","Type":"ContainerDied","Data":"4ae2ad9e287517d89a3b28b96de9747b5e09e8c941241a1b9ed2dd25bf358b29"}
Oct 30 19:30:04.501722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.501695    6777 scope.go:122] "RemoveContainer" containerID="4ae2ad9e287517d89a3b28b96de9747b5e09e8c941241a1b9ed2dd25bf358b29"
Oct 30 19:30:04.503734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.503713    6777 generic.go:356] "Generic (PLEG): container finished" podID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerID="e349cbf1b6288754a07734a6df8555918e4825a062bfb02365dcf17561016271" exitCode=60
Oct 30 19:30:04.503875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.503855    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" event={"ID":"31fa5b2a-01b1-4cba-b726-cd9332da7b4a","Type":"ContainerDied","Data":"e349cbf1b6288754a07734a6df8555918e4825a062bfb02365dcf17561016271"}
Oct 30 19:30:04.504142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.504122    6777 scope.go:122] "RemoveContainer" containerID="e349cbf1b6288754a07734a6df8555918e4825a062bfb02365dcf17561016271"
Oct 30 19:30:04.506570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.506545    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" event={"ID":"be08e6ae-ea90-4735-b444-211571ace69f","Type":"ContainerStarted","Data":"0191ece687b33ecf02d3af35e3602ec8184d2b568aa79959da73459ce2c329c1"}
Oct 30 19:30:04.594015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.593954    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" podStartSLOduration=4.5939342530000005 podStartE2EDuration="4.593934253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:04.590564638 +0000 UTC m=+1386.587480022" watchObservedRunningTime="2026-10-30 19:30:04.593934253 +0000 UTC m=+1386.590849650"
Oct 30 19:30:04.702118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:04.699845    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" podStartSLOduration=4.699827526 podStartE2EDuration="4.699827526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:04.661126827 +0000 UTC m=+1386.658042214" watchObservedRunningTime="2026-10-30 19:30:04.699827526 +0000 UTC m=+1386.696742950"
Oct 30 19:30:05.365130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.359693    6777 patch_prober.go:29] interesting pod/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:05.365130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.359764    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 19:30:05.519605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.519511    6777 generic.go:356] "Generic (PLEG): container finished" podID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerID="8aaf001d9e93f8f0240fcb2d8ad1cbc708abab2c78de39471cda7c0a0481efe4" exitCode=60
Oct 30 19:30:05.519605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.519585    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" event={"ID":"173b9bfc-03f9-4604-b5b5-3f2d0625d231","Type":"ContainerDied","Data":"8aaf001d9e93f8f0240fcb2d8ad1cbc708abab2c78de39471cda7c0a0481efe4"}
Oct 30 19:30:05.520044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.519652    6777 scope.go:122] "RemoveContainer" containerID="3bea8cd9e157c5f13f8f0f709ecad6219bc3c119ba770c61d0f3209b167e67f1"
Oct 30 19:30:05.520044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.519948    6777 scope.go:122] "RemoveContainer" containerID="8aaf001d9e93f8f0240fcb2d8ad1cbc708abab2c78de39471cda7c0a0481efe4"
Oct 30 19:30:05.520464 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:05.520436    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4_odh-ai-gateway-infra(173b9bfc-03f9-4604-b5b5-3f2d0625d231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231"
Oct 30 19:30:05.521952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.521929    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" event={"ID":"1276cd7c-e245-4663-a46f-d03854c463c7","Type":"ContainerStarted","Data":"ff5bcfeff4375a3108a53a61898c1071ce8700c6a6edd4989d87ed67845a04e1"}
Oct 30 19:30:05.524233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.524207    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" event={"ID":"31fa5b2a-01b1-4cba-b726-cd9332da7b4a","Type":"ContainerStarted","Data":"818449ef48b9b4b1d5da1ca1a1c2ff09ae6182a4b3dcd72716eda51aa4be48a0"}
Oct 30 19:30:05.526452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.526427    6777 generic.go:356] "Generic (PLEG): container finished" podID="be08e6ae-ea90-4735-b444-211571ace69f" containerID="0191ece687b33ecf02d3af35e3602ec8184d2b568aa79959da73459ce2c329c1" exitCode=60
Oct 30 19:30:05.526541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.526516    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" event={"ID":"be08e6ae-ea90-4735-b444-211571ace69f","Type":"ContainerDied","Data":"0191ece687b33ecf02d3af35e3602ec8184d2b568aa79959da73459ce2c329c1"}
Oct 30 19:30:05.526710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.526695    6777 scope.go:122] "RemoveContainer" containerID="0191ece687b33ecf02d3af35e3602ec8184d2b568aa79959da73459ce2c329c1"
Oct 30 19:30:05.527050 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:05.527030    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd_odh-ai-gateway-infra(be08e6ae-ea90-4735-b444-211571ace69f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" podUID="be08e6ae-ea90-4735-b444-211571ace69f"
Oct 30 19:30:05.530013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.529980    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" event={"ID":"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c","Type":"ContainerStarted","Data":"2ae76831036cbd0a82918a37b9d70c8d4982049e1485755623803674bd0cfccf"}
Oct 30 19:30:05.536902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.536875    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" event={"ID":"569c8ef0-8451-4faa-895f-f4f497143b94","Type":"ContainerStarted","Data":"461b23089c0ac745d0f8ab99026d9272c3eeedaaa38898d71365fa2cb7a9572e"}
Oct 30 19:30:05.540760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.540723    6777 scope.go:122] "RemoveContainer" containerID="965e7160a51840fc7a972bd796c471d853e7fb487c8baa4d9a722cb2bd70b7ea"
Oct 30 19:30:05.644145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.644086    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" podStartSLOduration=5.644071444 podStartE2EDuration="5.644071444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.641524281 +0000 UTC m=+1387.638439667" watchObservedRunningTime="2026-10-30 19:30:05.644071444 +0000 UTC m=+1387.640986820"
Oct 30 19:30:05.704554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.704501    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" podStartSLOduration=5.704487581 podStartE2EDuration="5.704487581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.700461388 +0000 UTC m=+1387.697376772" watchObservedRunningTime="2026-10-30 19:30:05.704487581 +0000 UTC m=+1387.701403015"
Oct 30 19:30:05.705813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.705775    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" podStartSLOduration=5.705763919 podStartE2EDuration="5.705763919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.672796085 +0000 UTC m=+1387.669711472" watchObservedRunningTime="2026-10-30 19:30:05.705763919 +0000 UTC m=+1387.702679304"
Oct 30 19:30:05.723615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.723576    6777 patch_prober.go:29] interesting pod/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:30:05.723798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:05.723643    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:30:06.365237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.365206    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:30:06.365753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.365731    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:30:06.366675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.366655    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:30:06.551004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.550971    6777 scope.go:122] "RemoveContainer" containerID="8aaf001d9e93f8f0240fcb2d8ad1cbc708abab2c78de39471cda7c0a0481efe4"
Oct 30 19:30:06.551558 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:06.551426    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4_odh-ai-gateway-infra(173b9bfc-03f9-4604-b5b5-3f2d0625d231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231"
Oct 30 19:30:06.553704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.553674    6777 generic.go:356] "Generic (PLEG): container finished" podID="1276cd7c-e245-4663-a46f-d03854c463c7" containerID="ff5bcfeff4375a3108a53a61898c1071ce8700c6a6edd4989d87ed67845a04e1" exitCode=60
Oct 30 19:30:06.553833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.553721    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" event={"ID":"1276cd7c-e245-4663-a46f-d03854c463c7","Type":"ContainerDied","Data":"ff5bcfeff4375a3108a53a61898c1071ce8700c6a6edd4989d87ed67845a04e1"}
Oct 30 19:30:06.553833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.553795    6777 scope.go:122] "RemoveContainer" containerID="4ae2ad9e287517d89a3b28b96de9747b5e09e8c941241a1b9ed2dd25bf358b29"
Oct 30 19:30:06.554159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.554144    6777 scope.go:122] "RemoveContainer" containerID="ff5bcfeff4375a3108a53a61898c1071ce8700c6a6edd4989d87ed67845a04e1"
Oct 30 19:30:06.554592 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:06.554566    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s_odh-ai-gateway-infra(1276cd7c-e245-4663-a46f-d03854c463c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" podUID="1276cd7c-e245-4663-a46f-d03854c463c7"
Oct 30 19:30:06.556182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.556157    6777 generic.go:356] "Generic (PLEG): container finished" podID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerID="818449ef48b9b4b1d5da1ca1a1c2ff09ae6182a4b3dcd72716eda51aa4be48a0" exitCode=60
Oct 30 19:30:06.556286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.556262    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" event={"ID":"31fa5b2a-01b1-4cba-b726-cd9332da7b4a","Type":"ContainerDied","Data":"818449ef48b9b4b1d5da1ca1a1c2ff09ae6182a4b3dcd72716eda51aa4be48a0"}
Oct 30 19:30:06.556595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.556575    6777 scope.go:122] "RemoveContainer" containerID="818449ef48b9b4b1d5da1ca1a1c2ff09ae6182a4b3dcd72716eda51aa4be48a0"
Oct 30 19:30:06.557047 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:06.557019    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj_odh-ai-gateway-infra(31fa5b2a-01b1-4cba-b726-cd9332da7b4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a"
Oct 30 19:30:06.558592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.558573    6777 scope.go:122] "RemoveContainer" containerID="0191ece687b33ecf02d3af35e3602ec8184d2b568aa79959da73459ce2c329c1"
Oct 30 19:30:06.558952 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:06.558928    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd_odh-ai-gateway-infra(be08e6ae-ea90-4735-b444-211571ace69f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" podUID="be08e6ae-ea90-4735-b444-211571ace69f"
Oct 30 19:30:06.560687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.560664    6777 generic.go:356] "Generic (PLEG): container finished" podID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerID="2ae76831036cbd0a82918a37b9d70c8d4982049e1485755623803674bd0cfccf" exitCode=60
Oct 30 19:30:06.560799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.560757    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" event={"ID":"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c","Type":"ContainerDied","Data":"2ae76831036cbd0a82918a37b9d70c8d4982049e1485755623803674bd0cfccf"}
Oct 30 19:30:06.561073 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.561041    6777 scope.go:122] "RemoveContainer" containerID="2ae76831036cbd0a82918a37b9d70c8d4982049e1485755623803674bd0cfccf"
Oct 30 19:30:06.561698 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:06.561675    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq_odh-ai-gateway-infra(dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c"
Oct 30 19:30:06.563699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.563677    6777 generic.go:356] "Generic (PLEG): container finished" podID="569c8ef0-8451-4faa-895f-f4f497143b94" containerID="461b23089c0ac745d0f8ab99026d9272c3eeedaaa38898d71365fa2cb7a9572e" exitCode=60
Oct 30 19:30:06.563813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.563771    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" event={"ID":"569c8ef0-8451-4faa-895f-f4f497143b94","Type":"ContainerDied","Data":"461b23089c0ac745d0f8ab99026d9272c3eeedaaa38898d71365fa2cb7a9572e"}
Oct 30 19:30:06.564020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.564002    6777 scope.go:122] "RemoveContainer" containerID="461b23089c0ac745d0f8ab99026d9272c3eeedaaa38898d71365fa2cb7a9572e"
Oct 30 19:30:06.564359 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:06.564333    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc_odh-ai-gateway-infra(569c8ef0-8451-4faa-895f-f4f497143b94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" podUID="569c8ef0-8451-4faa-895f-f4f497143b94"
Oct 30 19:30:06.569946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.569919    6777 scope.go:122] "RemoveContainer" containerID="e349cbf1b6288754a07734a6df8555918e4825a062bfb02365dcf17561016271"
Oct 30 19:30:06.585551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.585532    6777 scope.go:122] "RemoveContainer" containerID="038493e75bafdeb3952ed81bf5dfd6fa65cf6e4bb56a3dad3bebaa6c5c4b6d5c"
Oct 30 19:30:06.600845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.600823    6777 scope.go:122] "RemoveContainer" containerID="91e31016f05890a7cc03bb2bce4e74911c30d53bdf43670400803924b65cb1cd"
Oct 30 19:30:06.741871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.741778    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"]
Oct 30 19:30:06.749206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.749181    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.765948 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.765923    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-33b4d7\"/\"kube-root-ca.crt\""
Oct 30 19:30:06.765948 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.765936    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-33b4d7\"/\"openshift-service-ca.crt\""
Oct 30 19:30:06.766601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.766573    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-model-cache\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.766732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.766678    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdb4\" (UniqueName: \"kubernetes.io/projected/76810fef-bb01-431d-b4a9-faf73e8f25ce-kube-api-access-khdb4\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.766814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.766768    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-tmp-dir\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.766959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.766933    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76810fef-bb01-431d-b4a9-faf73e8f25ce-tls-certs\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.767026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.766999    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-home\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.767183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.767165    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-33b4d7\"/\"default-dockercfg-fx7nd\""
Oct 30 19:30:06.767291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.767272    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-dshm\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.780106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.776141    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-33b4d7\"/\"test-model-33b4d7-kserve-self-signed-certs\""
Oct 30 19:30:06.785498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.785472    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"]
Oct 30 19:30:06.869364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.869325    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-tmp-dir\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.869539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.869452    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76810fef-bb01-431d-b4a9-faf73e8f25ce-tls-certs\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.869539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.869495    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-home\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.869747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.869725    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-dshm\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.869832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.869814    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-tmp-dir\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.869898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.869828    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-model-cache\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.869898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.869850    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-home\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.870008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.869913    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdb4\" (UniqueName: \"kubernetes.io/projected/76810fef-bb01-431d-b4a9-faf73e8f25ce-kube-api-access-khdb4\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.870237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.870214    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-model-cache\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.872194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.872164    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-dshm\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.872415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.872396    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76810fef-bb01-431d-b4a9-faf73e8f25ce-tls-certs\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:06.922954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:06.920237    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdb4\" (UniqueName: \"kubernetes.io/projected/76810fef-bb01-431d-b4a9-faf73e8f25ce-kube-api-access-khdb4\") pod \"test-model-33b4d7-kserve-6d995b4df-4qmhf\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") " pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:07.063661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:07.063626    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:07.658145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:07.658108    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"]
Oct 30 19:30:07.672790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:07.672758    6777 scope.go:122] "RemoveContainer" containerID="ff5bcfeff4375a3108a53a61898c1071ce8700c6a6edd4989d87ed67845a04e1"
Oct 30 19:30:07.673371 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:07.673321    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s_odh-ai-gateway-infra(1276cd7c-e245-4663-a46f-d03854c463c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" podUID="1276cd7c-e245-4663-a46f-d03854c463c7"
Oct 30 19:30:07.674983 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:07.674941    6777 scope.go:122] "RemoveContainer" containerID="818449ef48b9b4b1d5da1ca1a1c2ff09ae6182a4b3dcd72716eda51aa4be48a0"
Oct 30 19:30:07.675395 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:07.675371    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj_odh-ai-gateway-infra(31fa5b2a-01b1-4cba-b726-cd9332da7b4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a"
Oct 30 19:30:07.677746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:07.677725    6777 scope.go:122] "RemoveContainer" containerID="2ae76831036cbd0a82918a37b9d70c8d4982049e1485755623803674bd0cfccf"
Oct 30 19:30:07.678913 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:07.678885    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq_odh-ai-gateway-infra(dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c"
Oct 30 19:30:07.681295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:07.681273    6777 scope.go:122] "RemoveContainer" containerID="461b23089c0ac745d0f8ab99026d9272c3eeedaaa38898d71365fa2cb7a9572e"
Oct 30 19:30:07.681651 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:07.681622    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc_odh-ai-gateway-infra(569c8ef0-8451-4faa-895f-f4f497143b94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" podUID="569c8ef0-8451-4faa-895f-f4f497143b94"
Oct 30 19:30:07.684401 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:30:07.684374    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76810fef_bb01_431d_b4a9_faf73e8f25ce.slice/crio-11dc8d2785d6060a67497f036c5ff08b36f8d597e15d9130c96117023a3f5fc0 WatchSource:0}: Error finding container 11dc8d2785d6060a67497f036c5ff08b36f8d597e15d9130c96117023a3f5fc0: Status 404 returned error can't find the container with id 11dc8d2785d6060a67497f036c5ff08b36f8d597e15d9130c96117023a3f5fc0
Oct 30 19:30:08.322258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.322231    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:30:08.364042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.363995    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"]
Oct 30 19:30:08.365392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.365363    6777 cpu_manager.go:414] "containerMap: removing container" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy"
Oct 30 19:30:08.365392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.365391    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy"
Oct 30 19:30:08.365641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.365626    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy"
Oct 30 19:30:08.365704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.365647    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy"
Oct 30 19:30:08.371806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.371667    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"]
Oct 30 19:30:08.372457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.371771    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:08.376934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.376890    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-02feaa\""
Oct 30 19:30:08.382509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.382482    6777 cpu_manager.go:414] "containerMap: removing container" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy"
Oct 30 19:30:08.382509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.382509    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969296cc-bcee-4ad7-8944-977164eac29c" containerName="istio-proxy"
Oct 30 19:30:08.387329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.387304    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"]
Oct 30 19:30:08.387506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.387428    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:08.390772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.390747    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"]
Oct 30 19:30:08.408545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.408519    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-02feaa-dockercfg-qrbgf\""
Oct 30 19:30:08.408665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.408554    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ef5dca-dockercfg-74cps\""
Oct 30 19:30:08.430790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.429141    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/969296cc-bcee-4ad7-8944-977164eac29c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969296cc-bcee-4ad7-8944-977164eac29c-istio-podinfo\") pod \"969296cc-bcee-4ad7-8944-977164eac29c\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") "
Oct 30 19:30:08.430790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.429232    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-certs\") pod \"969296cc-bcee-4ad7-8944-977164eac29c\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") "
Oct 30 19:30:08.430790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.429359    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-kube-api-access-gncvw\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-kube-api-access-gncvw\") pod \"969296cc-bcee-4ad7-8944-977164eac29c\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") "
Oct 30 19:30:08.430790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.429442    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-socket\") pod \"969296cc-bcee-4ad7-8944-977164eac29c\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") "
Oct 30 19:30:08.430790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.429691    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969296cc-bcee-4ad7-8944-977164eac29c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969296cc-bcee-4ad7-8944-977164eac29c-istiod-ca-cert\") pod \"969296cc-bcee-4ad7-8944-977164eac29c\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") "
Oct 30 19:30:08.430790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.429770    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-istio-token\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-istio-token\") pod \"969296cc-bcee-4ad7-8944-977164eac29c\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") "
Oct 30 19:30:08.430790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.429825    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-credential-socket\") pod \"969296cc-bcee-4ad7-8944-977164eac29c\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") "
Oct 30 19:30:08.430790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.429921    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-envoy\") pod \"969296cc-bcee-4ad7-8944-977164eac29c\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") "
Oct 30 19:30:08.430790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.429961    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-data\") pod \"969296cc-bcee-4ad7-8944-977164eac29c\" (UID: \"969296cc-bcee-4ad7-8944-977164eac29c\") "
Oct 30 19:30:08.432148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.432075    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-data" pod "969296cc-bcee-4ad7-8944-977164eac29c" (UID: "969296cc-bcee-4ad7-8944-977164eac29c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:08.432798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.432773    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969296cc-bcee-4ad7-8944-977164eac29c-istiod-ca-cert" pod "969296cc-bcee-4ad7-8944-977164eac29c" (UID: "969296cc-bcee-4ad7-8944-977164eac29c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:08.433219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.433196    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-certs" pod "969296cc-bcee-4ad7-8944-977164eac29c" (UID: "969296cc-bcee-4ad7-8944-977164eac29c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:08.433818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.433797    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-socket" pod "969296cc-bcee-4ad7-8944-977164eac29c" (UID: "969296cc-bcee-4ad7-8944-977164eac29c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:08.434142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.434119    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-credential-socket" pod "969296cc-bcee-4ad7-8944-977164eac29c" (UID: "969296cc-bcee-4ad7-8944-977164eac29c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:08.462203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.455953    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-istio-token" pod "969296cc-bcee-4ad7-8944-977164eac29c" (UID: "969296cc-bcee-4ad7-8944-977164eac29c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:08.473211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.471336    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/969296cc-bcee-4ad7-8944-977164eac29c-istio-podinfo" pod "969296cc-bcee-4ad7-8944-977164eac29c" (UID: "969296cc-bcee-4ad7-8944-977164eac29c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:08.473211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.471495    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-envoy" pod "969296cc-bcee-4ad7-8944-977164eac29c" (UID: "969296cc-bcee-4ad7-8944-977164eac29c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:08.477454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.477341    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-kube-api-access-gncvw" pod "969296cc-bcee-4ad7-8944-977164eac29c" (UID: "969296cc-bcee-4ad7-8944-977164eac29c"). InnerVolumeSpecName "kube-api-access-gncvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:08.478568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.478544    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ef5dca\""
Oct 30 19:30:08.532932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.532893    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5114c4-03c6-480b-baea-4f8657ff6a58-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4\" (UID: \"de5114c4-03c6-480b-baea-4f8657ff6a58\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:08.533334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533284    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2hc\" (UniqueName: \"kubernetes.io/projected/7d475743-3ee9-452a-bd4c-6de427d0a778-kube-api-access-zr2hc\") pod \"payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j\" (UID: \"7d475743-3ee9-452a-bd4c-6de427d0a778\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:08.533454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533355    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbxmm\" (UniqueName: \"kubernetes.io/projected/de5114c4-03c6-480b-baea-4f8657ff6a58-kube-api-access-dbxmm\") pod \"payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4\" (UID: \"de5114c4-03c6-480b-baea-4f8657ff6a58\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:08.533454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533404    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d475743-3ee9-452a-bd4c-6de427d0a778-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j\" (UID: \"7d475743-3ee9-452a-bd4c-6de427d0a778\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:08.533582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533568    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:08.533625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533588    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:08.533625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533606    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:08.533625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533621    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:08.533737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533634    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969296cc-bcee-4ad7-8944-977164eac29c-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:08.533737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533648    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:08.533737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533664    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gncvw\" (UniqueName: \"kubernetes.io/projected/969296cc-bcee-4ad7-8944-977164eac29c-kube-api-access-gncvw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:08.533737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533678    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969296cc-bcee-4ad7-8944-977164eac29c-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:08.533737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.533693    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969296cc-bcee-4ad7-8944-977164eac29c-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:08.635154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.635116    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2hc\" (UniqueName: \"kubernetes.io/projected/7d475743-3ee9-452a-bd4c-6de427d0a778-kube-api-access-zr2hc\") pod \"payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j\" (UID: \"7d475743-3ee9-452a-bd4c-6de427d0a778\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:08.635338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.635169    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxmm\" (UniqueName: \"kubernetes.io/projected/de5114c4-03c6-480b-baea-4f8657ff6a58-kube-api-access-dbxmm\") pod \"payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4\" (UID: \"de5114c4-03c6-480b-baea-4f8657ff6a58\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:08.635338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.635217    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d475743-3ee9-452a-bd4c-6de427d0a778-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j\" (UID: \"7d475743-3ee9-452a-bd4c-6de427d0a778\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:08.635461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.635412    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5114c4-03c6-480b-baea-4f8657ff6a58-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4\" (UID: \"de5114c4-03c6-480b-baea-4f8657ff6a58\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:08.636029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.636003    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d475743-3ee9-452a-bd4c-6de427d0a778-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j\" (UID: \"7d475743-3ee9-452a-bd4c-6de427d0a778\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:08.636174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.636118    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5114c4-03c6-480b-baea-4f8657ff6a58-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4\" (UID: \"de5114c4-03c6-480b-baea-4f8657ff6a58\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:08.657997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.657963    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2hc\" (UniqueName: \"kubernetes.io/projected/7d475743-3ee9-452a-bd4c-6de427d0a778-kube-api-access-zr2hc\") pod \"payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j\" (UID: \"7d475743-3ee9-452a-bd4c-6de427d0a778\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:08.658156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.658048    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxmm\" (UniqueName: \"kubernetes.io/projected/de5114c4-03c6-480b-baea-4f8657ff6a58-kube-api-access-dbxmm\") pod \"payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4\" (UID: \"de5114c4-03c6-480b-baea-4f8657ff6a58\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:08.692752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.692714    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf" event={"ID":"76810fef-bb01-431d-b4a9-faf73e8f25ce","Type":"ContainerStarted","Data":"b99b72adbf06c7ed0771d891c815317512c53fbd18fe706bbe953b0bf8278427"}
Oct 30 19:30:08.693197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.692766    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf" event={"ID":"76810fef-bb01-431d-b4a9-faf73e8f25ce","Type":"ContainerStarted","Data":"11dc8d2785d6060a67497f036c5ff08b36f8d597e15d9130c96117023a3f5fc0"}
Oct 30 19:30:08.694889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.694860    6777 generic.go:356] "Generic (PLEG): container finished" podID="969296cc-bcee-4ad7-8944-977164eac29c" containerID="e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847" exitCode=0
Oct 30 19:30:08.695038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.694912    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" event={"ID":"969296cc-bcee-4ad7-8944-977164eac29c","Type":"ContainerDied","Data":"e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847"}
Oct 30 19:30:08.695038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.694929    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"
Oct 30 19:30:08.695038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.694950    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r" event={"ID":"969296cc-bcee-4ad7-8944-977164eac29c","Type":"ContainerDied","Data":"05d026a14432b9aa95c18e1e321577d1cec3786918ccd8176d0ab952317771b1"}
Oct 30 19:30:08.695038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.694985    6777 scope.go:122] "RemoveContainer" containerID="e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847"
Oct 30 19:30:08.704037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.703964    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:08.708100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.708076    6777 scope.go:122] "RemoveContainer" containerID="c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca"
Oct 30 19:30:08.712355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.712296    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:08.718798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.718747    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf" podStartSLOduration=2.7186982779999997 podStartE2EDuration="2.718698278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:08.712998546 +0000 UTC m=+1390.709913933" watchObservedRunningTime="2026-10-30 19:30:08.718698278 +0000 UTC m=+1390.715613664"
Oct 30 19:30:08.724269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.724242    6777 scope.go:122] "RemoveContainer" containerID="e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847"
Oct 30 19:30:08.724654 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:08.724625    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847\": container with ID starting with e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847 not found: ID does not exist" containerID="e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847"
Oct 30 19:30:08.724740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.724664    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847"} err="failed to get container status \"e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847\": rpc error: code = NotFound desc = could not find container \"e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847\": container with ID starting with e6b64e70f4d25ddb677b67afa134c0cbf98e3141befadaa8035d5fe9d96be847 not found: ID does not exist"
Oct 30 19:30:08.724740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.724690    6777 scope.go:122] "RemoveContainer" containerID="c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca"
Oct 30 19:30:08.725036 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:08.725014    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca\": container with ID starting with c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca not found: ID does not exist" containerID="c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca"
Oct 30 19:30:08.725180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.725043    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca"} err="failed to get container status \"c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca\": rpc error: code = NotFound desc = could not find container \"c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca\": container with ID starting with c9a20bbdf5b61da57a636c6882f4a2632d174da44003ec5931dc04f00b5572ca not found: ID does not exist"
Oct 30 19:30:08.741830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.741794    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"]
Oct 30 19:30:08.748382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:08.748355    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-48b1a6a7-openshift-default-78f6598f5c-br66r"]
Oct 30 19:30:09.036160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.036119    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"]
Oct 30 19:30:09.064446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.064409    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:09.091556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.091525    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"]
Oct 30 19:30:09.095841 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:30:09.095814    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d475743_3ee9_452a_bd4c_6de427d0a778.slice/crio-dc76ffe9940f5fdc9236c2420e179be30850e20d30417600fb50874237234e91 WatchSource:0}: Error finding container dc76ffe9940f5fdc9236c2420e179be30850e20d30417600fb50874237234e91: Status 404 returned error can't find the container with id dc76ffe9940f5fdc9236c2420e179be30850e20d30417600fb50874237234e91
Oct 30 19:30:09.705974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.705930    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4" event={"ID":"de5114c4-03c6-480b-baea-4f8657ff6a58","Type":"ContainerStarted","Data":"7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457"}
Oct 30 19:30:09.706485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.706002    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4" event={"ID":"de5114c4-03c6-480b-baea-4f8657ff6a58","Type":"ContainerStarted","Data":"8147a907695796fcd835d247877909e6999d48cc23464826365fb38bc47ffb14"}
Oct 30 19:30:09.706485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.706033    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:09.707878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.707853    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j" event={"ID":"7d475743-3ee9-452a-bd4c-6de427d0a778","Type":"ContainerStarted","Data":"c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747"}
Oct 30 19:30:09.708017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.707903    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j" event={"ID":"7d475743-3ee9-452a-bd4c-6de427d0a778","Type":"ContainerStarted","Data":"dc76ffe9940f5fdc9236c2420e179be30850e20d30417600fb50874237234e91"}
Oct 30 19:30:09.708017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.707949    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:09.732498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.732449    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4" podStartSLOduration=8.732435957 podStartE2EDuration="8.732435957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:09.726933485 +0000 UTC m=+1391.723848871" watchObservedRunningTime="2026-10-30 19:30:09.732435957 +0000 UTC m=+1391.729351344"
Oct 30 19:30:09.753025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:09.752972    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j" podStartSLOduration=2.752958673 podStartE2EDuration="2.752958673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:09.74803892 +0000 UTC m=+1391.744954307" watchObservedRunningTime="2026-10-30 19:30:09.752958673 +0000 UTC m=+1391.749874086"
Oct 30 19:30:10.116677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:10.116646    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969296cc-bcee-4ad7-8944-977164eac29c" path="/var/lib/kubelet/pods/969296cc-bcee-4ad7-8944-977164eac29c/volumes"
Oct 30 19:30:13.642647 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:13.642609    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:30:17.072567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:17.072531    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:17.117086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:17.106209    6777 scope.go:122] "RemoveContainer" containerID="461b23089c0ac745d0f8ab99026d9272c3eeedaaa38898d71365fa2cb7a9572e"
Oct 30 19:30:17.117086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:17.107822    6777 scope.go:122] "RemoveContainer" containerID="0191ece687b33ecf02d3af35e3602ec8184d2b568aa79959da73459ce2c329c1"
Oct 30 19:30:17.230504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:17.230408    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:17.802284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:17.802242    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" event={"ID":"be08e6ae-ea90-4735-b444-211571ace69f","Type":"ContainerStarted","Data":"725ca868cf6e2d2c6b374f939086ffb3b6b1721ee36afff590da57dde4d10cdf"}
Oct 30 19:30:17.805954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:17.805919    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" event={"ID":"569c8ef0-8451-4faa-895f-f4f497143b94","Type":"ContainerStarted","Data":"fb460a181081ea4acb56a7f7a3ec4edbeceac186240af1fd7bf2fb0c37336bd3"}
Oct 30 19:30:17.821244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:17.821215    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:30:18.110649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.110614    6777 scope.go:122] "RemoveContainer" containerID="2ae76831036cbd0a82918a37b9d70c8d4982049e1485755623803674bd0cfccf"
Oct 30 19:30:18.111346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.110785    6777 scope.go:122] "RemoveContainer" containerID="8aaf001d9e93f8f0240fcb2d8ad1cbc708abab2c78de39471cda7c0a0481efe4"
Oct 30 19:30:18.111346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.111003    6777 scope.go:122] "RemoveContainer" containerID="818449ef48b9b4b1d5da1ca1a1c2ff09ae6182a4b3dcd72716eda51aa4be48a0"
Oct 30 19:30:18.826289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.826245    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" event={"ID":"31fa5b2a-01b1-4cba-b726-cd9332da7b4a","Type":"ContainerStarted","Data":"ac20bf976172bcb55f4070f3ba497734fd02a48a634dad229a1563aa4f773790"}
Oct 30 19:30:18.829161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.829134    6777 generic.go:356] "Generic (PLEG): container finished" podID="be08e6ae-ea90-4735-b444-211571ace69f" containerID="725ca868cf6e2d2c6b374f939086ffb3b6b1721ee36afff590da57dde4d10cdf" exitCode=60
Oct 30 19:30:18.829308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.829196    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" event={"ID":"be08e6ae-ea90-4735-b444-211571ace69f","Type":"ContainerDied","Data":"725ca868cf6e2d2c6b374f939086ffb3b6b1721ee36afff590da57dde4d10cdf"}
Oct 30 19:30:18.829308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.829254    6777 scope.go:122] "RemoveContainer" containerID="0191ece687b33ecf02d3af35e3602ec8184d2b568aa79959da73459ce2c329c1"
Oct 30 19:30:18.829725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.829702    6777 scope.go:122] "RemoveContainer" containerID="725ca868cf6e2d2c6b374f939086ffb3b6b1721ee36afff590da57dde4d10cdf"
Oct 30 19:30:18.830211 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:18.830185    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd_odh-ai-gateway-infra(be08e6ae-ea90-4735-b444-211571ace69f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" podUID="be08e6ae-ea90-4735-b444-211571ace69f"
Oct 30 19:30:18.833000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.832977    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" event={"ID":"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c","Type":"ContainerStarted","Data":"5d411e71f1a3a45e62b76fbcda8b2fa2bb13bd6a7a7d87785fd9c9502d26e915"}
Oct 30 19:30:18.836498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.836466    6777 generic.go:356] "Generic (PLEG): container finished" podID="569c8ef0-8451-4faa-895f-f4f497143b94" containerID="fb460a181081ea4acb56a7f7a3ec4edbeceac186240af1fd7bf2fb0c37336bd3" exitCode=60
Oct 30 19:30:18.836619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.836550    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" event={"ID":"569c8ef0-8451-4faa-895f-f4f497143b94","Type":"ContainerDied","Data":"fb460a181081ea4acb56a7f7a3ec4edbeceac186240af1fd7bf2fb0c37336bd3"}
Oct 30 19:30:18.837040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.837016    6777 scope.go:122] "RemoveContainer" containerID="fb460a181081ea4acb56a7f7a3ec4edbeceac186240af1fd7bf2fb0c37336bd3"
Oct 30 19:30:18.837514 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:18.837489    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc_odh-ai-gateway-infra(569c8ef0-8451-4faa-895f-f4f497143b94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" podUID="569c8ef0-8451-4faa-895f-f4f497143b94"
Oct 30 19:30:18.839232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.839196    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" event={"ID":"173b9bfc-03f9-4604-b5b5-3f2d0625d231","Type":"ContainerStarted","Data":"ec030fd8bdbcacbe60b09e2adb35096bc12081308cbcc11f2a23523742789b5e"}
Oct 30 19:30:18.840046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.840019    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"]
Oct 30 19:30:18.853621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.853594    6777 scope.go:122] "RemoveContainer" containerID="461b23089c0ac745d0f8ab99026d9272c3eeedaaa38898d71365fa2cb7a9572e"
Oct 30 19:30:18.888421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:18.888392    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"]
Oct 30 19:30:19.105877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.105847    6777 scope.go:122] "RemoveContainer" containerID="ff5bcfeff4375a3108a53a61898c1071ce8700c6a6edd4989d87ed67845a04e1"
Oct 30 19:30:19.866300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.866253    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"]
Oct 30 19:30:19.870570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.870390    6777 generic.go:356] "Generic (PLEG): container finished" podID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerID="ec030fd8bdbcacbe60b09e2adb35096bc12081308cbcc11f2a23523742789b5e" exitCode=60
Oct 30 19:30:19.870570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.870503    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" event={"ID":"173b9bfc-03f9-4604-b5b5-3f2d0625d231","Type":"ContainerDied","Data":"ec030fd8bdbcacbe60b09e2adb35096bc12081308cbcc11f2a23523742789b5e"}
Oct 30 19:30:19.870570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.870564    6777 scope.go:122] "RemoveContainer" containerID="8aaf001d9e93f8f0240fcb2d8ad1cbc708abab2c78de39471cda7c0a0481efe4"
Oct 30 19:30:19.871271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.871248    6777 scope.go:122] "RemoveContainer" containerID="ec030fd8bdbcacbe60b09e2adb35096bc12081308cbcc11f2a23523742789b5e"
Oct 30 19:30:19.871768 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:19.871740    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4_odh-ai-gateway-infra(173b9bfc-03f9-4604-b5b5-3f2d0625d231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231"
Oct 30 19:30:19.873873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.873847    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" event={"ID":"1276cd7c-e245-4663-a46f-d03854c463c7","Type":"ContainerStarted","Data":"42e3cb38fc14f5d0b3b08c068b13f45064cbd25e35fc2b29b0134c38a9d02cbb"}
Oct 30 19:30:19.876332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.876304    6777 generic.go:356] "Generic (PLEG): container finished" podID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerID="ac20bf976172bcb55f4070f3ba497734fd02a48a634dad229a1563aa4f773790" exitCode=60
Oct 30 19:30:19.876438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.876381    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" event={"ID":"31fa5b2a-01b1-4cba-b726-cd9332da7b4a","Type":"ContainerDied","Data":"ac20bf976172bcb55f4070f3ba497734fd02a48a634dad229a1563aa4f773790"}
Oct 30 19:30:19.886109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.881600    6777 generic.go:356] "Generic (PLEG): container finished" podID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerID="5d411e71f1a3a45e62b76fbcda8b2fa2bb13bd6a7a7d87785fd9c9502d26e915" exitCode=60
Oct 30 19:30:19.886109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.881705    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" event={"ID":"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c","Type":"ContainerDied","Data":"5d411e71f1a3a45e62b76fbcda8b2fa2bb13bd6a7a7d87785fd9c9502d26e915"}
Oct 30 19:30:19.886109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.885521    6777 scope.go:122] "RemoveContainer" containerID="5d411e71f1a3a45e62b76fbcda8b2fa2bb13bd6a7a7d87785fd9c9502d26e915"
Oct 30 19:30:19.886109 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:19.886000    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq_odh-ai-gateway-infra(dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c"
Oct 30 19:30:19.891173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.891085    6777 scope.go:122] "RemoveContainer" containerID="818449ef48b9b4b1d5da1ca1a1c2ff09ae6182a4b3dcd72716eda51aa4be48a0"
Oct 30 19:30:19.894511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.894484    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"]
Oct 30 19:30:19.930881 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.930845    6777 scope.go:122] "RemoveContainer" containerID="2ae76831036cbd0a82918a37b9d70c8d4982049e1485755623803674bd0cfccf"
Oct 30 19:30:19.931116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:19.931090    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"]
Oct 30 19:30:20.127553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.127526    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:20.218021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.217982    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:20.236399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.236376    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:20.256936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.256902    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-kube-api-access-fs2rd\" (UniqueName: \"kubernetes.io/projected/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-kube-api-access-fs2rd\") pod \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\" (UID: \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\") "
Oct 30 19:30:20.257122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.256986    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-maas-api-service-ca\") pod \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\" (UID: \"31fa5b2a-01b1-4cba-b726-cd9332da7b4a\") "
Oct 30 19:30:20.257623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.257579    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-maas-api-service-ca" pod "31fa5b2a-01b1-4cba-b726-cd9332da7b4a" (UID: "31fa5b2a-01b1-4cba-b726-cd9332da7b4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:20.257888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.257853    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.260379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.260357    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-kube-api-access-fs2rd" pod "31fa5b2a-01b1-4cba-b726-cd9332da7b4a" (UID: "31fa5b2a-01b1-4cba-b726-cd9332da7b4a"). InnerVolumeSpecName "kube-api-access-fs2rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:20.358610 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.358570    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be08e6ae-ea90-4735-b444-211571ace69f-kube-api-access-6d8vf\" (UniqueName: \"kubernetes.io/projected/be08e6ae-ea90-4735-b444-211571ace69f-kube-api-access-6d8vf\") pod \"be08e6ae-ea90-4735-b444-211571ace69f\" (UID: \"be08e6ae-ea90-4735-b444-211571ace69f\") "
Oct 30 19:30:20.358728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.358704    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569c8ef0-8451-4faa-895f-f4f497143b94-kube-api-access-vnf6h\" (UniqueName: \"kubernetes.io/projected/569c8ef0-8451-4faa-895f-f4f497143b94-kube-api-access-vnf6h\") pod \"569c8ef0-8451-4faa-895f-f4f497143b94\" (UID: \"569c8ef0-8451-4faa-895f-f4f497143b94\") "
Oct 30 19:30:20.358996 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.358958    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569c8ef0-8451-4faa-895f-f4f497143b94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8ef0-8451-4faa-895f-f4f497143b94-maas-api-service-ca\") pod \"569c8ef0-8451-4faa-895f-f4f497143b94\" (UID: \"569c8ef0-8451-4faa-895f-f4f497143b94\") "
Oct 30 19:30:20.359481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.359165    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be08e6ae-ea90-4735-b444-211571ace69f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be08e6ae-ea90-4735-b444-211571ace69f-maas-api-service-ca\") pod \"be08e6ae-ea90-4735-b444-211571ace69f\" (UID: \"be08e6ae-ea90-4735-b444-211571ace69f\") "
Oct 30 19:30:20.359481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.359390    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c8ef0-8451-4faa-895f-f4f497143b94-maas-api-service-ca" pod "569c8ef0-8451-4faa-895f-f4f497143b94" (UID: "569c8ef0-8451-4faa-895f-f4f497143b94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:20.359481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.359447    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be08e6ae-ea90-4735-b444-211571ace69f-maas-api-service-ca" pod "be08e6ae-ea90-4735-b444-211571ace69f" (UID: "be08e6ae-ea90-4735-b444-211571ace69f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:20.360101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.360037    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8ef0-8451-4faa-895f-f4f497143b94-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.360203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.360110    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be08e6ae-ea90-4735-b444-211571ace69f-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.360203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.360127    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs2rd\" (UniqueName: \"kubernetes.io/projected/31fa5b2a-01b1-4cba-b726-cd9332da7b4a-kube-api-access-fs2rd\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.361342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.361305    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be08e6ae-ea90-4735-b444-211571ace69f-kube-api-access-6d8vf" pod "be08e6ae-ea90-4735-b444-211571ace69f" (UID: "be08e6ae-ea90-4735-b444-211571ace69f"). InnerVolumeSpecName "kube-api-access-6d8vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:20.362392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.362366    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569c8ef0-8451-4faa-895f-f4f497143b94-kube-api-access-vnf6h" pod "569c8ef0-8451-4faa-895f-f4f497143b94" (UID: "569c8ef0-8451-4faa-895f-f4f497143b94"). InnerVolumeSpecName "kube-api-access-vnf6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:20.462658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.462623    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d8vf\" (UniqueName: \"kubernetes.io/projected/be08e6ae-ea90-4735-b444-211571ace69f-kube-api-access-6d8vf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.462658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.462660    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnf6h\" (UniqueName: \"kubernetes.io/projected/569c8ef0-8451-4faa-895f-f4f497143b94-kube-api-access-vnf6h\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.563261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.563224    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"]
Oct 30 19:30:20.564336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564311    6777 cpu_manager.go:414] "containerMap: removing container" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" containerName="cleanup"
Oct 30 19:30:20.564336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564332    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564344    6777 cpu_manager.go:414] "containerMap: removing container" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564350    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564372    6777 cpu_manager.go:414] "containerMap: removing container" podUID="be08e6ae-ea90-4735-b444-211571ace69f" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564377    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be08e6ae-ea90-4735-b444-211571ace69f" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564390    6777 cpu_manager.go:414] "containerMap: removing container" podUID="be08e6ae-ea90-4735-b444-211571ace69f" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564395    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be08e6ae-ea90-4735-b444-211571ace69f" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564401    6777 cpu_manager.go:414] "containerMap: removing container" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564406    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564433    6777 cpu_manager.go:414] "containerMap: removing container" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564437    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564448    6777 cpu_manager.go:414] "containerMap: removing container" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerName="cleanup"
Oct 30 19:30:20.564530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564455    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerName="cleanup"
Oct 30 19:30:20.564889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564569    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="be08e6ae-ea90-4735-b444-211571ace69f" containerName="cleanup"
Oct 30 19:30:20.564889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564578    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="be08e6ae-ea90-4735-b444-211571ace69f" containerName="cleanup"
Oct 30 19:30:20.564889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564585    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerName="cleanup"
Oct 30 19:30:20.564889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564591    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" containerName="cleanup"
Oct 30 19:30:20.564889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564600    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="be08e6ae-ea90-4735-b444-211571ace69f" containerName="cleanup"
Oct 30 19:30:20.564889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564607    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" containerName="cleanup"
Oct 30 19:30:20.564889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.564614    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerName="cleanup"
Oct 30 19:30:20.568609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.568590    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:20.575711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.575684    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d52364\""
Oct 30 19:30:20.579995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.579971    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"]
Oct 30 19:30:20.666421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.666328    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsj9t\" (UniqueName: \"kubernetes.io/projected/1d962588-5797-46dd-9413-e4c2efcf21f8-kube-api-access-nsj9t\") pod \"payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh\" (UID: \"1d962588-5797-46dd-9413-e4c2efcf21f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:20.666421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.666399    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d962588-5797-46dd-9413-e4c2efcf21f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh\" (UID: \"1d962588-5797-46dd-9413-e4c2efcf21f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:20.771083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.770425    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:30:20.778090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.773366    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsj9t\" (UniqueName: \"kubernetes.io/projected/1d962588-5797-46dd-9413-e4c2efcf21f8-kube-api-access-nsj9t\") pod \"payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh\" (UID: \"1d962588-5797-46dd-9413-e4c2efcf21f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:20.778090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.773423    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d962588-5797-46dd-9413-e4c2efcf21f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh\" (UID: \"1d962588-5797-46dd-9413-e4c2efcf21f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:20.795608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.792970    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d962588-5797-46dd-9413-e4c2efcf21f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh\" (UID: \"1d962588-5797-46dd-9413-e4c2efcf21f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:20.795608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.793294    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:30:20.886548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.882851    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsj9t\" (UniqueName: \"kubernetes.io/projected/1d962588-5797-46dd-9413-e4c2efcf21f8-kube-api-access-nsj9t\") pod \"payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh\" (UID: \"1d962588-5797-46dd-9413-e4c2efcf21f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:20.895368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:20.891363    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:21.068962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.068859    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"]
Oct 30 19:30:21.116419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.116378    6777 generic.go:356] "Generic (PLEG): container finished" podID="1276cd7c-e245-4663-a46f-d03854c463c7" containerID="42e3cb38fc14f5d0b3b08c068b13f45064cbd25e35fc2b29b0134c38a9d02cbb" exitCode=60
Oct 30 19:30:21.117495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.117463    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" event={"ID":"1276cd7c-e245-4663-a46f-d03854c463c7","Type":"ContainerDied","Data":"42e3cb38fc14f5d0b3b08c068b13f45064cbd25e35fc2b29b0134c38a9d02cbb"}
Oct 30 19:30:21.119778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.119757    6777 scope.go:122] "RemoveContainer" containerID="ff5bcfeff4375a3108a53a61898c1071ce8700c6a6edd4989d87ed67845a04e1"
Oct 30 19:30:21.171151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.157704    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj" event={"ID":"31fa5b2a-01b1-4cba-b726-cd9332da7b4a","Type":"ContainerDied","Data":"79ea0414a900cd68cdc0676435b7e02817914407693584234a5dbe3adb24fec0"}
Oct 30 19:30:21.171151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.157866    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"
Oct 30 19:30:21.171151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.168977    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd" event={"ID":"be08e6ae-ea90-4735-b444-211571ace69f","Type":"ContainerDied","Data":"7d9b883ef63c0f1bdb2085a8d970b9fb329584ec054d67df5eaac903876e2366"}
Oct 30 19:30:21.171151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.169165    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"
Oct 30 19:30:21.183245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.180601    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc" event={"ID":"569c8ef0-8451-4faa-895f-f4f497143b94","Type":"ContainerDied","Data":"59aafb22a0a9c2c169c0b291e345bfcc5c01fbc19b9e052395fefc6cec290495"}
Oct 30 19:30:21.183245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.180765    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"
Oct 30 19:30:21.227765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.227642    6777 scope.go:122] "RemoveContainer" containerID="ac20bf976172bcb55f4070f3ba497734fd02a48a634dad229a1563aa4f773790"
Oct 30 19:30:21.256087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.254928    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"]
Oct 30 19:30:21.263317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.262920    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889810-jgqrj"]
Oct 30 19:30:21.268341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.268309    6777 scope.go:122] "RemoveContainer" containerID="725ca868cf6e2d2c6b374f939086ffb3b6b1721ee36afff590da57dde4d10cdf"
Oct 30 19:30:21.276589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.276558    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"]
Oct 30 19:30:21.284830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.284807    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889810-ljtdd"]
Oct 30 19:30:21.293778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.293745    6777 scope.go:122] "RemoveContainer" containerID="fb460a181081ea4acb56a7f7a3ec4edbeceac186240af1fd7bf2fb0c37336bd3"
Oct 30 19:30:21.305461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.305398    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"]
Oct 30 19:30:21.309213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.309054    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889810-mmnsc"]
Oct 30 19:30:21.624532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.624505    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:21.630048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.630020    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:21.634568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.634545    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:21.644252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.644220    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"]
Oct 30 19:30:21.732513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.732483    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/173b9bfc-03f9-4604-b5b5-3f2d0625d231-kube-api-access-jtmhm\" (UniqueName: \"kubernetes.io/projected/173b9bfc-03f9-4604-b5b5-3f2d0625d231-kube-api-access-jtmhm\") pod \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\" (UID: \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\") "
Oct 30 19:30:21.732660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.732597    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1276cd7c-e245-4663-a46f-d03854c463c7-kube-api-access-bpxt5\" (UniqueName: \"kubernetes.io/projected/1276cd7c-e245-4663-a46f-d03854c463c7-kube-api-access-bpxt5\") pod \"1276cd7c-e245-4663-a46f-d03854c463c7\" (UID: \"1276cd7c-e245-4663-a46f-d03854c463c7\") "
Oct 30 19:30:21.732718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.732660    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1276cd7c-e245-4663-a46f-d03854c463c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276cd7c-e245-4663-a46f-d03854c463c7-maas-api-service-ca\") pod \"1276cd7c-e245-4663-a46f-d03854c463c7\" (UID: \"1276cd7c-e245-4663-a46f-d03854c463c7\") "
Oct 30 19:30:21.732836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.732808    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/173b9bfc-03f9-4604-b5b5-3f2d0625d231-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/173b9bfc-03f9-4604-b5b5-3f2d0625d231-maas-api-service-ca\") pod \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\" (UID: \"173b9bfc-03f9-4604-b5b5-3f2d0625d231\") "
Oct 30 19:30:21.732909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.732892    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-kube-api-access-ggbhz\" (UniqueName: \"kubernetes.io/projected/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-kube-api-access-ggbhz\") pod \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\" (UID: \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\") "
Oct 30 19:30:21.733050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.733020    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1276cd7c-e245-4663-a46f-d03854c463c7-maas-api-service-ca" pod "1276cd7c-e245-4663-a46f-d03854c463c7" (UID: "1276cd7c-e245-4663-a46f-d03854c463c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:21.733161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.733140    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-maas-api-service-ca\") pod \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\" (UID: \"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c\") "
Oct 30 19:30:21.733382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.733358    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173b9bfc-03f9-4604-b5b5-3f2d0625d231-maas-api-service-ca" pod "173b9bfc-03f9-4604-b5b5-3f2d0625d231" (UID: "173b9bfc-03f9-4604-b5b5-3f2d0625d231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:21.734027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.733992    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-maas-api-service-ca" pod "dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" (UID: "dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:21.734181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.734163    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:21.734239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.734185    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276cd7c-e245-4663-a46f-d03854c463c7-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:21.734239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.734200    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/173b9bfc-03f9-4604-b5b5-3f2d0625d231-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:21.748603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.748549    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173b9bfc-03f9-4604-b5b5-3f2d0625d231-kube-api-access-jtmhm" pod "173b9bfc-03f9-4604-b5b5-3f2d0625d231" (UID: "173b9bfc-03f9-4604-b5b5-3f2d0625d231"). InnerVolumeSpecName "kube-api-access-jtmhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:21.748743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.748662    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-kube-api-access-ggbhz" pod "dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" (UID: "dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c"). InnerVolumeSpecName "kube-api-access-ggbhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:21.750955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.750921    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1276cd7c-e245-4663-a46f-d03854c463c7-kube-api-access-bpxt5" pod "1276cd7c-e245-4663-a46f-d03854c463c7" (UID: "1276cd7c-e245-4663-a46f-d03854c463c7"). InnerVolumeSpecName "kube-api-access-bpxt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:21.836150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.836110    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtmhm\" (UniqueName: \"kubernetes.io/projected/173b9bfc-03f9-4604-b5b5-3f2d0625d231-kube-api-access-jtmhm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:21.836150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.836145    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpxt5\" (UniqueName: \"kubernetes.io/projected/1276cd7c-e245-4663-a46f-d03854c463c7-kube-api-access-bpxt5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:21.836150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:21.836156    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggbhz\" (UniqueName: \"kubernetes.io/projected/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c-kube-api-access-ggbhz\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:22.117754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.117719    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" path="/var/lib/kubelet/pods/31fa5b2a-01b1-4cba-b726-cd9332da7b4a/volumes"
Oct 30 19:30:22.118539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.118518    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" path="/var/lib/kubelet/pods/569c8ef0-8451-4faa-895f-f4f497143b94/volumes"
Oct 30 19:30:22.119046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.119030    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be08e6ae-ea90-4735-b444-211571ace69f" path="/var/lib/kubelet/pods/be08e6ae-ea90-4735-b444-211571ace69f/volumes"
Oct 30 19:30:22.213716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.212227    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh" event={"ID":"1d962588-5797-46dd-9413-e4c2efcf21f8","Type":"ContainerStarted","Data":"1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479"}
Oct 30 19:30:22.213716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.212300    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh" event={"ID":"1d962588-5797-46dd-9413-e4c2efcf21f8","Type":"ContainerStarted","Data":"4d99124b9ff362f51f87ce450d4eaa8e86b5402a5abadca245a2e5f82cef44b7"}
Oct 30 19:30:22.213716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.213654    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:22.223476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.216496    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"
Oct 30 19:30:22.223476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.223318    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4" event={"ID":"173b9bfc-03f9-4604-b5b5-3f2d0625d231","Type":"ContainerDied","Data":"55598d931309d35f18cb7d781e5511548d1e28bb395012cd4f1ff4e066ea2320"}
Oct 30 19:30:22.223476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.223409    6777 scope.go:122] "RemoveContainer" containerID="ec030fd8bdbcacbe60b09e2adb35096bc12081308cbcc11f2a23523742789b5e"
Oct 30 19:30:22.235023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.231491    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"
Oct 30 19:30:22.235023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.231523    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s" event={"ID":"1276cd7c-e245-4663-a46f-d03854c463c7","Type":"ContainerDied","Data":"723d60594723e68af8e0a75967cb2ba62c7eb0086b2fab5f8c60dbed861da703"}
Oct 30 19:30:22.251805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.238279    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq" event={"ID":"dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c","Type":"ContainerDied","Data":"53b0322b935f984419ccb7d52b2fe468a7e3b64185247560611b899896bb01e2"}
Oct 30 19:30:22.251805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.238390    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"
Oct 30 19:30:22.274716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.274629    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh" podStartSLOduration=2.274598999 podStartE2EDuration="2.274598999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:22.249530684 +0000 UTC m=+1404.246446070" watchObservedRunningTime="2026-10-30 19:30:22.274598999 +0000 UTC m=+1404.271514415"
Oct 30 19:30:22.279188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.276232    6777 scope.go:122] "RemoveContainer" containerID="42e3cb38fc14f5d0b3b08c068b13f45064cbd25e35fc2b29b0134c38a9d02cbb"
Oct 30 19:30:22.284152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.284105    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"]
Oct 30 19:30:22.305574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.304168    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-27b5d5-29889810-zrm5s"]
Oct 30 19:30:22.319350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.316328    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"]
Oct 30 19:30:22.319350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.318870    6777 scope.go:122] "RemoveContainer" containerID="5d411e71f1a3a45e62b76fbcda8b2fa2bb13bd6a7a7d87785fd9c9502d26e915"
Oct 30 19:30:22.324122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.324029    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889810-8jkr4"]
Oct 30 19:30:22.329392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.329361    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"]
Oct 30 19:30:22.334363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:22.334334    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bce71f-29889810-ghkkq"]
Oct 30 19:30:24.117338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:24.117303    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" path="/var/lib/kubelet/pods/1276cd7c-e245-4663-a46f-d03854c463c7/volumes"
Oct 30 19:30:24.117811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:24.117796    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" path="/var/lib/kubelet/pods/173b9bfc-03f9-4604-b5b5-3f2d0625d231/volumes"
Oct 30 19:30:24.118274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:24.118259    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" path="/var/lib/kubelet/pods/dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c/volumes"
Oct 30 19:30:24.805012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:24.804975    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:30:29.528083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:29.527369    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.542210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:29.542161    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.822709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:29.822646    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf" podUID="76810fef-bb01-431d-b4a9-faf73e8f25ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.938473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:29.911334    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.957034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:29.956988    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.257480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:30.256150    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.305084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:30.293486    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.613106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:30.594785    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.753654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:30.753617    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:30:30.822585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:30.821382    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:30:34.282041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:34.282011    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:30:35.848221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:35.848174    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"]
Oct 30 19:30:35.848655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:35.848437    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy" containerID="cri-o://c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5" gracePeriod=30
Oct 30 19:30:36.140137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:36.140012    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"]
Oct 30 19:30:36.140308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:36.140238    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy" containerID="cri-o://ce1dfb49eff11602c1ce71724b00d87b36314ea43145b0de31b63324bd78dc4b" gracePeriod=30
Oct 30 19:30:36.147136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:36.147099    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"]
Oct 30 19:30:36.147365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:36.147341    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy" containerID="cri-o://5bce58a4a5b13594139691ffaaa21d8d081a6aad30f422bae98ae9fb23265142" gracePeriod=30
Oct 30 19:30:36.153290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:36.153260    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"]
Oct 30 19:30:36.153487 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:36.153464    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy" containerID="cri-o://e48ef53d1197e7fc4b2dbcfdfde524b3e3cc476a4ab43a55354b4b674c9ade9a" gracePeriod=30
Oct 30 19:30:36.561219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:36.561184    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"]
Oct 30 19:30:36.561488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:36.561437    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy" containerID="cri-o://a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265" gracePeriod=30
Oct 30 19:30:40.190864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.190830    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"]
Oct 30 19:30:40.191722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191703    6777 cpu_manager.go:414] "containerMap: removing container" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerName="cleanup"
Oct 30 19:30:40.191768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191724    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerName="cleanup"
Oct 30 19:30:40.191768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191748    6777 cpu_manager.go:414] "containerMap: removing container" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerName="cleanup"
Oct 30 19:30:40.191768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191756    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191769    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191775    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191794    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191800    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191811    6777 cpu_manager.go:414] "containerMap: removing container" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191817    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191843    6777 cpu_manager.go:414] "containerMap: removing container" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191851    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191858    6777 cpu_manager.go:414] "containerMap: removing container" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerName="cleanup"
Oct 30 19:30:40.191865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191865    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191874    6777 cpu_manager.go:414] "containerMap: removing container" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191879    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191893    6777 cpu_manager.go:414] "containerMap: removing container" podUID="be08e6ae-ea90-4735-b444-211571ace69f" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.191897    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be08e6ae-ea90-4735-b444-211571ace69f" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.192016    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.192025    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="569c8ef0-8451-4faa-895f-f4f497143b94" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.192035    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.192041    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fa5b2a-01b1-4cba-b726-cd9332da7b4a" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.192048    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.192069    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.192076    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.192084    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerName="cleanup"
Oct 30 19:30:40.192187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.192089    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerName="cleanup"
Oct 30 19:30:40.197872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.197850    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.210498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.210473    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:30:40.210638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.210586    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:30:40.213132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.213109    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"]
Oct 30 19:30:40.387393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.387363    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.387552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.387404    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.387691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.387669    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m44h\" (UniqueName: \"kubernetes.io/projected/0d8c65cf-f75b-40d5-85e4-f07d93b82995-kube-api-access-2m44h\") pod \"maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.488971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.488885    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.488971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.488927    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.489190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.489145    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m44h\" (UniqueName: \"kubernetes.io/projected/0d8c65cf-f75b-40d5-85e4-f07d93b82995-kube-api-access-2m44h\") pod \"maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.489652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.489629    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.491387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.491368    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.503127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.503103    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m44h\" (UniqueName: \"kubernetes.io/projected/0d8c65cf-f75b-40d5-85e4-f07d93b82995-kube-api-access-2m44h\") pod \"maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.517158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.517130    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:40.807074 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:30:40.807029    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8c65cf_f75b_40d5_85e4_f07d93b82995.slice/crio-a19e4081380e2d1f6a4b0916204f9d2af9bc3d80d9a8f9869ed1d1ec157369be WatchSource:0}: Error finding container a19e4081380e2d1f6a4b0916204f9d2af9bc3d80d9a8f9869ed1d1ec157369be: Status 404 returned error can't find the container with id a19e4081380e2d1f6a4b0916204f9d2af9bc3d80d9a8f9869ed1d1ec157369be
Oct 30 19:30:40.807224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:40.807044    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"]
Oct 30 19:30:41.323934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.323904    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:30:41.596112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.583397    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-podinfo\") pod \"77f1534e-4438-4789-aeee-f944b3dd15fc\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") "
Oct 30 19:30:41.596112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.583456    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-envoy\") pod \"77f1534e-4438-4789-aeee-f944b3dd15fc\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") "
Oct 30 19:30:41.596112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.583534    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-credential-socket\") pod \"77f1534e-4438-4789-aeee-f944b3dd15fc\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") "
Oct 30 19:30:41.596112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.583626    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-socket\") pod \"77f1534e-4438-4789-aeee-f944b3dd15fc\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") "
Oct 30 19:30:41.596112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.583765    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-data\") pod \"77f1534e-4438-4789-aeee-f944b3dd15fc\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") "
Oct 30 19:30:41.596112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.583801    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-certs\") pod \"77f1534e-4438-4789-aeee-f944b3dd15fc\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") "
Oct 30 19:30:41.596112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.583868    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-kube-api-access-g7xm5\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-kube-api-access-g7xm5\") pod \"77f1534e-4438-4789-aeee-f944b3dd15fc\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") "
Oct 30 19:30:41.596112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.583952    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f1534e-4438-4789-aeee-f944b3dd15fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f1534e-4438-4789-aeee-f944b3dd15fc-istiod-ca-cert\") pod \"77f1534e-4438-4789-aeee-f944b3dd15fc\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") "
Oct 30 19:30:41.596112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.584031    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-token\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-token\") pod \"77f1534e-4438-4789-aeee-f944b3dd15fc\" (UID: \"77f1534e-4438-4789-aeee-f944b3dd15fc\") "
Oct 30 19:30:41.729273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.669853    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-credential-socket" pod "77f1534e-4438-4789-aeee-f944b3dd15fc" (UID: "77f1534e-4438-4789-aeee-f944b3dd15fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:41.729273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.702958    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-certs" pod "77f1534e-4438-4789-aeee-f944b3dd15fc" (UID: "77f1534e-4438-4789-aeee-f944b3dd15fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:41.729273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.703186    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-socket" pod "77f1534e-4438-4789-aeee-f944b3dd15fc" (UID: "77f1534e-4438-4789-aeee-f944b3dd15fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:41.729273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.703345    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-data" pod "77f1534e-4438-4789-aeee-f944b3dd15fc" (UID: "77f1534e-4438-4789-aeee-f944b3dd15fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:41.763178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.760926    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-kube-api-access-g7xm5" pod "77f1534e-4438-4789-aeee-f944b3dd15fc" (UID: "77f1534e-4438-4789-aeee-f944b3dd15fc"). InnerVolumeSpecName "kube-api-access-g7xm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:41.763178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.761344    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f1534e-4438-4789-aeee-f944b3dd15fc-istiod-ca-cert" pod "77f1534e-4438-4789-aeee-f944b3dd15fc" (UID: "77f1534e-4438-4789-aeee-f944b3dd15fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:41.837255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.831178    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.837255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.831214    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.837255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.831226    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.837255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.831238    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.837255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.831250    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xm5\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-kube-api-access-g7xm5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.837255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.831263    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f1534e-4438-4789-aeee-f944b3dd15fc-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.882362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.877249    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-token" pod "77f1534e-4438-4789-aeee-f944b3dd15fc" (UID: "77f1534e-4438-4789-aeee-f944b3dd15fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:41.929960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.896329    6777 generic.go:356] "Generic (PLEG): container finished" podID="6449741c-acf1-41b6-80dd-e3416a04a548" containerID="5bce58a4a5b13594139691ffaaa21d8d081a6aad30f422bae98ae9fb23265142" exitCode=0
Oct 30 19:30:41.929960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.896564    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" event={"ID":"6449741c-acf1-41b6-80dd-e3416a04a548","Type":"ContainerDied","Data":"5bce58a4a5b13594139691ffaaa21d8d081a6aad30f422bae98ae9fb23265142"}
Oct 30 19:30:41.929960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.899912    6777 generic.go:356] "Generic (PLEG): container finished" podID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerID="c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5" exitCode=0
Oct 30 19:30:41.929960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.900042    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" event={"ID":"77f1534e-4438-4789-aeee-f944b3dd15fc","Type":"ContainerDied","Data":"c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5"}
Oct 30 19:30:41.929960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.900113    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw" event={"ID":"77f1534e-4438-4789-aeee-f944b3dd15fc","Type":"ContainerDied","Data":"ccaea8c28c3e51a90cbca1470340c936c327b56d1d7d8638075a299903aac990"}
Oct 30 19:30:41.929960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.900146    6777 scope.go:122] "RemoveContainer" containerID="c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5"
Oct 30 19:30:41.929960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.900337    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"
Oct 30 19:30:41.938223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.936316    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-podinfo" pod "77f1534e-4438-4789-aeee-f944b3dd15fc" (UID: "77f1534e-4438-4789-aeee-f944b3dd15fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:41.938223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.938042    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.938223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.938133    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.965537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.956203    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-envoy" pod "77f1534e-4438-4789-aeee-f944b3dd15fc" (UID: "77f1534e-4438-4789-aeee-f944b3dd15fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:41.998211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:41.997990    6777 scope.go:122] "RemoveContainer" containerID="c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5"
Oct 30 19:30:42.018825 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:42.018779    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5\": container with ID starting with c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5 not found: ID does not exist" containerID="c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5"
Oct 30 19:30:42.019119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.019083    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5"} err="failed to get container status \"c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5\": rpc error: code = NotFound desc = could not find container \"c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5\": container with ID starting with c0191539de7c177aa2ca5b1cccf2b0de2d1a44f221ae59f0c800a0b9ee85f8e5 not found: ID does not exist"
Oct 30 19:30:42.019809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.019782    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7" event={"ID":"0d8c65cf-f75b-40d5-85e4-f07d93b82995","Type":"ContainerStarted","Data":"a19e4081380e2d1f6a4b0916204f9d2af9bc3d80d9a8f9869ed1d1ec157369be"}
Oct 30 19:30:42.024946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.024739    6777 generic.go:356] "Generic (PLEG): container finished" podID="4959594f-0d2e-49ed-a21a-17db03609138" containerID="e48ef53d1197e7fc4b2dbcfdfde524b3e3cc476a4ab43a55354b4b674c9ade9a" exitCode=0
Oct 30 19:30:42.024946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.024832    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" event={"ID":"4959594f-0d2e-49ed-a21a-17db03609138","Type":"ContainerDied","Data":"e48ef53d1197e7fc4b2dbcfdfde524b3e3cc476a4ab43a55354b4b674c9ade9a"}
Oct 30 19:30:42.039305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.032993    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"]
Oct 30 19:30:42.039305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.038589    6777 generic.go:356] "Generic (PLEG): container finished" podID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerID="ce1dfb49eff11602c1ce71724b00d87b36314ea43145b0de31b63324bd78dc4b" exitCode=0
Oct 30 19:30:42.039305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.038740    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" event={"ID":"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3","Type":"ContainerDied","Data":"ce1dfb49eff11602c1ce71724b00d87b36314ea43145b0de31b63324bd78dc4b"}
Oct 30 19:30:42.040336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.040098    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f1534e-4438-4789-aeee-f944b3dd15fc-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.046930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.046903    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ceda0-nv9d9"]
Oct 30 19:30:42.050252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.050229    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:30:42.061959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.056554    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:30:42.124760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.122723    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a" path="/var/lib/kubelet/pods/40a0c8c2-a2f8-4d86-a09c-49b3e85d9b3a/volumes"
Oct 30 19:30:42.140823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.140795    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-data\") pod \"6449741c-acf1-41b6-80dd-e3416a04a548\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") "
Oct 30 19:30:42.140938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.140846    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-token\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-token\") pod \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") "
Oct 30 19:30:42.140938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.140897    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-istio-token\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-istio-token\") pod \"6449741c-acf1-41b6-80dd-e3416a04a548\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") "
Oct 30 19:30:42.141053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.140944    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-certs\") pod \"6449741c-acf1-41b6-80dd-e3416a04a548\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") "
Oct 30 19:30:42.141053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.140984    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-credential-socket\") pod \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") "
Oct 30 19:30:42.141053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141008    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-envoy\") pod \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") "
Oct 30 19:30:42.141053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141040    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-socket\") pod \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") "
Oct 30 19:30:42.141299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141140    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-data\") pod \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") "
Oct 30 19:30:42.141299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141187    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-credential-socket\") pod \"6449741c-acf1-41b6-80dd-e3416a04a548\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") "
Oct 30 19:30:42.141299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141206    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6449741c-acf1-41b6-80dd-e3416a04a548-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6449741c-acf1-41b6-80dd-e3416a04a548-istio-podinfo\") pod \"6449741c-acf1-41b6-80dd-e3416a04a548\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") "
Oct 30 19:30:42.141299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141252    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-kube-api-access-f965r\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-kube-api-access-f965r\") pod \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") "
Oct 30 19:30:42.141299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141287    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-podinfo\") pod \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") "
Oct 30 19:30:42.141462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141321    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-kube-api-access-d88p9\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-kube-api-access-d88p9\") pod \"6449741c-acf1-41b6-80dd-e3416a04a548\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") "
Oct 30 19:30:42.141462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141338    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-certs\") pod \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") "
Oct 30 19:30:42.141462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141381    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istiod-ca-cert\") pod \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\" (UID: \"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3\") "
Oct 30 19:30:42.141462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141400    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-socket\") pod \"6449741c-acf1-41b6-80dd-e3416a04a548\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") "
Oct 30 19:30:42.141586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141489    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-envoy\") pod \"6449741c-acf1-41b6-80dd-e3416a04a548\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") "
Oct 30 19:30:42.141586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.141560    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6449741c-acf1-41b6-80dd-e3416a04a548-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6449741c-acf1-41b6-80dd-e3416a04a548-istiod-ca-cert\") pod \"6449741c-acf1-41b6-80dd-e3416a04a548\" (UID: \"6449741c-acf1-41b6-80dd-e3416a04a548\") "
Oct 30 19:30:42.142408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.142389    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-certs" pod "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" (UID: "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.142572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.142422    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6449741c-acf1-41b6-80dd-e3416a04a548-istiod-ca-cert" pod "6449741c-acf1-41b6-80dd-e3416a04a548" (UID: "6449741c-acf1-41b6-80dd-e3416a04a548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:42.142777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.142752    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istiod-ca-cert" pod "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" (UID: "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:42.143043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.143019    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-socket" pod "6449741c-acf1-41b6-80dd-e3416a04a548" (UID: "6449741c-acf1-41b6-80dd-e3416a04a548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.143254 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.143232    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-data" pod "6449741c-acf1-41b6-80dd-e3416a04a548" (UID: "6449741c-acf1-41b6-80dd-e3416a04a548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.143349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.143329    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-token" pod "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" (UID: "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:42.143619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.143600    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-certs" pod "6449741c-acf1-41b6-80dd-e3416a04a548" (UID: "6449741c-acf1-41b6-80dd-e3416a04a548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.144833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.144383    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-socket" pod "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" (UID: "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.144833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.144577    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-credential-socket" pod "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" (UID: "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.145141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.145120    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6449741c-acf1-41b6-80dd-e3416a04a548-istio-podinfo" pod "6449741c-acf1-41b6-80dd-e3416a04a548" (UID: "6449741c-acf1-41b6-80dd-e3416a04a548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:42.145485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.145465    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-credential-socket" pod "6449741c-acf1-41b6-80dd-e3416a04a548" (UID: "6449741c-acf1-41b6-80dd-e3416a04a548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.146196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.146175    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-data" pod "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" (UID: "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.146611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.146595    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-istio-token" pod "6449741c-acf1-41b6-80dd-e3416a04a548" (UID: "6449741c-acf1-41b6-80dd-e3416a04a548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:42.154865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.154842    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:30:42.157747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.157714    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-envoy" pod "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" (UID: "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.181619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.181583    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-envoy" pod "6449741c-acf1-41b6-80dd-e3416a04a548" (UID: "6449741c-acf1-41b6-80dd-e3416a04a548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.181970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.181899    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-podinfo" pod "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" (UID: "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:42.182616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.182265    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-kube-api-access-d88p9" pod "6449741c-acf1-41b6-80dd-e3416a04a548" (UID: "6449741c-acf1-41b6-80dd-e3416a04a548"). InnerVolumeSpecName "kube-api-access-d88p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:42.184532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.184503    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-kube-api-access-f965r" pod "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" (UID: "44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3"). InnerVolumeSpecName "kube-api-access-f965r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:42.243493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.243461    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-istio-token\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-istio-token\") pod \"4959594f-0d2e-49ed-a21a-17db03609138\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") "
Oct 30 19:30:42.243691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.243531    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-envoy\") pod \"4959594f-0d2e-49ed-a21a-17db03609138\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") "
Oct 30 19:30:42.243691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.243569    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-credential-socket\") pod \"4959594f-0d2e-49ed-a21a-17db03609138\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") "
Oct 30 19:30:42.243691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.243651    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-data\") pod \"4959594f-0d2e-49ed-a21a-17db03609138\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") "
Oct 30 19:30:42.243815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.243706    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4959594f-0d2e-49ed-a21a-17db03609138-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4959594f-0d2e-49ed-a21a-17db03609138-istiod-ca-cert\") pod \"4959594f-0d2e-49ed-a21a-17db03609138\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") "
Oct 30 19:30:42.243815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.243796    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-certs\") pod \"4959594f-0d2e-49ed-a21a-17db03609138\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") "
Oct 30 19:30:42.243931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.243916    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-socket\") pod \"4959594f-0d2e-49ed-a21a-17db03609138\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") "
Oct 30 19:30:42.244012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.243990    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-kube-api-access-hk7h6\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-kube-api-access-hk7h6\") pod \"4959594f-0d2e-49ed-a21a-17db03609138\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") "
Oct 30 19:30:42.244163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.244037    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4959594f-0d2e-49ed-a21a-17db03609138-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4959594f-0d2e-49ed-a21a-17db03609138-istio-podinfo\") pod \"4959594f-0d2e-49ed-a21a-17db03609138\" (UID: \"4959594f-0d2e-49ed-a21a-17db03609138\") "
Oct 30 19:30:42.245001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.244978    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245002    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245013    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6449741c-acf1-41b6-80dd-e3416a04a548-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245023    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f965r\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-kube-api-access-f965r\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245033    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245042    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d88p9\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-kube-api-access-d88p9\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245052    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245079    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245092    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245105    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245117    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6449741c-acf1-41b6-80dd-e3416a04a548-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245132    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245144    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245157    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6449741c-acf1-41b6-80dd-e3416a04a548-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245171    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6449741c-acf1-41b6-80dd-e3416a04a548-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245186    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245198    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245212    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245384    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-data" pod "4959594f-0d2e-49ed-a21a-17db03609138" (UID: "4959594f-0d2e-49ed-a21a-17db03609138"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245513    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-certs" pod "4959594f-0d2e-49ed-a21a-17db03609138" (UID: "4959594f-0d2e-49ed-a21a-17db03609138"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.245653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245526    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-credential-socket" pod "4959594f-0d2e-49ed-a21a-17db03609138" (UID: "4959594f-0d2e-49ed-a21a-17db03609138"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.246184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245688    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-socket" pod "4959594f-0d2e-49ed-a21a-17db03609138" (UID: "4959594f-0d2e-49ed-a21a-17db03609138"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.246184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.245930    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4959594f-0d2e-49ed-a21a-17db03609138-istiod-ca-cert" pod "4959594f-0d2e-49ed-a21a-17db03609138" (UID: "4959594f-0d2e-49ed-a21a-17db03609138"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:42.247530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.247503    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-envoy" pod "4959594f-0d2e-49ed-a21a-17db03609138" (UID: "4959594f-0d2e-49ed-a21a-17db03609138"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.247662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.247642    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4959594f-0d2e-49ed-a21a-17db03609138-istio-podinfo" pod "4959594f-0d2e-49ed-a21a-17db03609138" (UID: "4959594f-0d2e-49ed-a21a-17db03609138"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:42.247740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.247726    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-istio-token" pod "4959594f-0d2e-49ed-a21a-17db03609138" (UID: "4959594f-0d2e-49ed-a21a-17db03609138"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:42.247778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.247758    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-kube-api-access-hk7h6" pod "4959594f-0d2e-49ed-a21a-17db03609138" (UID: "4959594f-0d2e-49ed-a21a-17db03609138"). InnerVolumeSpecName "kube-api-access-hk7h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:42.257402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.257375    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:30:42.266290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.266268    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"]
Oct 30 19:30:42.282812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.282782    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vn5jw"]
Oct 30 19:30:42.346025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.345996    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-token\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-token\") pod \"b56ca541-7f88-46cd-aa63-beb3988e72cd\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") "
Oct 30 19:30:42.346498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346050    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56ca541-7f88-46cd-aa63-beb3988e72cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ca541-7f88-46cd-aa63-beb3988e72cd-istiod-ca-cert\") pod \"b56ca541-7f88-46cd-aa63-beb3988e72cd\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") "
Oct 30 19:30:42.346498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346158    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-podinfo\") pod \"b56ca541-7f88-46cd-aa63-beb3988e72cd\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") "
Oct 30 19:30:42.346498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346216    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-certs\") pod \"b56ca541-7f88-46cd-aa63-beb3988e72cd\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") "
Oct 30 19:30:42.346498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346301    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-kube-api-access-8qrm8\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-kube-api-access-8qrm8\") pod \"b56ca541-7f88-46cd-aa63-beb3988e72cd\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") "
Oct 30 19:30:42.346498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346413    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-envoy\") pod \"b56ca541-7f88-46cd-aa63-beb3988e72cd\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") "
Oct 30 19:30:42.346785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346513    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-credential-socket\") pod \"b56ca541-7f88-46cd-aa63-beb3988e72cd\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") "
Oct 30 19:30:42.346785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346602    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-data\") pod \"b56ca541-7f88-46cd-aa63-beb3988e72cd\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") "
Oct 30 19:30:42.346785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346645    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-socket\") pod \"b56ca541-7f88-46cd-aa63-beb3988e72cd\" (UID: \"b56ca541-7f88-46cd-aa63-beb3988e72cd\") "
Oct 30 19:30:42.346785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346700    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56ca541-7f88-46cd-aa63-beb3988e72cd-istiod-ca-cert" pod "b56ca541-7f88-46cd-aa63-beb3988e72cd" (UID: "b56ca541-7f88-46cd-aa63-beb3988e72cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:42.347004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.346920    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-socket" pod "b56ca541-7f88-46cd-aa63-beb3988e72cd" (UID: "b56ca541-7f88-46cd-aa63-beb3988e72cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.347370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.347323    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-data" pod "b56ca541-7f88-46cd-aa63-beb3988e72cd" (UID: "b56ca541-7f88-46cd-aa63-beb3988e72cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.347724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.347702    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-credential-socket" pod "b56ca541-7f88-46cd-aa63-beb3988e72cd" (UID: "b56ca541-7f88-46cd-aa63-beb3988e72cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.348143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.348115    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-certs" pod "b56ca541-7f88-46cd-aa63-beb3988e72cd" (UID: "b56ca541-7f88-46cd-aa63-beb3988e72cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.348278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.348259    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.348392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.348377    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk7h6\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-kube-api-access-hk7h6\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.348483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.348468    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4959594f-0d2e-49ed-a21a-17db03609138-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.348611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.348596    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.348730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.348697    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4959594f-0d2e-49ed-a21a-17db03609138-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.349306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349291    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349449    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349093    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-token" pod "b56ca541-7f88-46cd-aa63-beb3988e72cd" (UID: "b56ca541-7f88-46cd-aa63-beb3988e72cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349249    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-envoy" pod "b56ca541-7f88-46cd-aa63-beb3988e72cd" (UID: "b56ca541-7f88-46cd-aa63-beb3988e72cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349256    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-kube-api-access-8qrm8" pod "b56ca541-7f88-46cd-aa63-beb3988e72cd" (UID: "b56ca541-7f88-46cd-aa63-beb3988e72cd"). InnerVolumeSpecName "kube-api-access-8qrm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349476    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349533    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349547    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349560    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4959594f-0d2e-49ed-a21a-17db03609138-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349570    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4959594f-0d2e-49ed-a21a-17db03609138-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.349578    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ca541-7f88-46cd-aa63-beb3988e72cd-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.351650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.350584    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-podinfo" pod "b56ca541-7f88-46cd-aa63-beb3988e72cd" (UID: "b56ca541-7f88-46cd-aa63-beb3988e72cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:42.451703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.451617    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.451703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.451650    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.451703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.451660    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.451703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.451672    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qrm8\" (UniqueName: \"kubernetes.io/projected/b56ca541-7f88-46cd-aa63-beb3988e72cd-kube-api-access-8qrm8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.451703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:42.451683    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ca541-7f88-46cd-aa63-beb3988e72cd-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:43.115713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.115671    6777 generic.go:356] "Generic (PLEG): container finished" podID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerID="a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265" exitCode=0
Oct 30 19:30:43.116099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.115903    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" event={"ID":"b56ca541-7f88-46cd-aa63-beb3988e72cd","Type":"ContainerDied","Data":"a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265"}
Oct 30 19:30:43.116099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.115952    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz" event={"ID":"b56ca541-7f88-46cd-aa63-beb3988e72cd","Type":"ContainerDied","Data":"dbaaf7896543a31a6b66e47d1b1fb317abf8d7aa01867fd35b6295864ea2852e"}
Oct 30 19:30:43.116099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.115984    6777 scope.go:122] "RemoveContainer" containerID="a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265"
Oct 30 19:30:43.116324 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.116182    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"
Oct 30 19:30:43.126393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.126315    6777 generic.go:356] "Generic (PLEG): container finished" podID="0d8c65cf-f75b-40d5-85e4-f07d93b82995" containerID="abfba7f2b790fae6c488f80c2474ae90b11f27826706213dbebe659e2b66034d" exitCode=0
Oct 30 19:30:43.127459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.126772    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7" event={"ID":"0d8c65cf-f75b-40d5-85e4-f07d93b82995","Type":"ContainerDied","Data":"abfba7f2b790fae6c488f80c2474ae90b11f27826706213dbebe659e2b66034d"}
Oct 30 19:30:43.151102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.138769    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj" event={"ID":"4959594f-0d2e-49ed-a21a-17db03609138","Type":"ContainerDied","Data":"062b4a9605dc6727d9df2d881f844db24ddbb6723e1785a355b1fa409d2fdd5d"}
Oct 30 19:30:43.151102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.138926    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"
Oct 30 19:30:43.163320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.162308    6777 scope.go:122] "RemoveContainer" containerID="a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265"
Oct 30 19:30:43.163554 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:43.163415    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265\": container with ID starting with a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265 not found: ID does not exist" containerID="a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265"
Oct 30 19:30:43.163554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.163454    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265"} err="failed to get container status \"a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265\": rpc error: code = NotFound desc = could not find container \"a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265\": container with ID starting with a598540f816c94313ad347847dcf10f19e56674ff5f336064c23bd6c68934265 not found: ID does not exist"
Oct 30 19:30:43.163554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.163479    6777 scope.go:122] "RemoveContainer" containerID="e48ef53d1197e7fc4b2dbcfdfde524b3e3cc476a4ab43a55354b4b674c9ade9a"
Oct 30 19:30:43.163729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.163602    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"
Oct 30 19:30:43.163729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.163626    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7" event={"ID":"44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3","Type":"ContainerDied","Data":"986f7fd8515f6bce29c3b714333573d1d98fc9f9b7d1f1824b0f04e149660357"}
Oct 30 19:30:43.165399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.165380    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4" event={"ID":"6449741c-acf1-41b6-80dd-e3416a04a548","Type":"ContainerDied","Data":"4e824bd9650dda3029b7b34dc2ee74c268c4f7b0b04aaf74fdeb32789b2427ca"}
Oct 30 19:30:43.165490 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.165461    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"
Oct 30 19:30:43.177511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.177394    6777 scope.go:122] "RemoveContainer" containerID="ce1dfb49eff11602c1ce71724b00d87b36314ea43145b0de31b63324bd78dc4b"
Oct 30 19:30:43.190814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.190744    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"]
Oct 30 19:30:43.196394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.196364    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-26psz"]
Oct 30 19:30:43.202747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.202691    6777 scope.go:122] "RemoveContainer" containerID="5bce58a4a5b13594139691ffaaa21d8d081a6aad30f422bae98ae9fb23265142"
Oct 30 19:30:43.218841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.218610    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"]
Oct 30 19:30:43.225646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.225617    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-sqxlj"]
Oct 30 19:30:43.251163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.251087    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"]
Oct 30 19:30:43.260023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.259982    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-rfjv7"]
Oct 30 19:30:43.276825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.276785    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"]
Oct 30 19:30:43.279863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:43.279829    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-mhkw4"]
Oct 30 19:30:43.656279 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:43.656199    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:30:44.123425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.123371    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" path="/var/lib/kubelet/pods/44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3/volumes"
Oct 30 19:30:44.127885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.124511    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4959594f-0d2e-49ed-a21a-17db03609138" path="/var/lib/kubelet/pods/4959594f-0d2e-49ed-a21a-17db03609138/volumes"
Oct 30 19:30:44.128702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.128515    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" path="/var/lib/kubelet/pods/6449741c-acf1-41b6-80dd-e3416a04a548/volumes"
Oct 30 19:30:44.129393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.129365    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" path="/var/lib/kubelet/pods/77f1534e-4438-4789-aeee-f944b3dd15fc/volumes"
Oct 30 19:30:44.130266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.130235    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" path="/var/lib/kubelet/pods/b56ca541-7f88-46cd-aa63-beb3988e72cd/volumes"
Oct 30 19:30:44.426875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.426845    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:44.545040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.545008    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8c65cf-f75b-40d5-85e4-f07d93b82995-kube-api-access-2m44h\" (UniqueName: \"kubernetes.io/projected/0d8c65cf-f75b-40d5-85e4-f07d93b82995-kube-api-access-2m44h\") pod \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") "
Oct 30 19:30:44.545201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.545094    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-openshift-ca\") pod \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") "
Oct 30 19:30:44.545366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.545335    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-xks-ca\") pod \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\" (UID: \"0d8c65cf-f75b-40d5-85e4-f07d93b82995\") "
Oct 30 19:30:44.545517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.545462    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-openshift-ca" pod "0d8c65cf-f75b-40d5-85e4-f07d93b82995" (UID: "0d8c65cf-f75b-40d5-85e4-f07d93b82995"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:44.546208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.546186    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:44.547342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.547318    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8c65cf-f75b-40d5-85e4-f07d93b82995-kube-api-access-2m44h" pod "0d8c65cf-f75b-40d5-85e4-f07d93b82995" (UID: "0d8c65cf-f75b-40d5-85e4-f07d93b82995"). InnerVolumeSpecName "kube-api-access-2m44h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:44.547497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.547477    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-xks-ca" pod "0d8c65cf-f75b-40d5-85e4-f07d93b82995" (UID: "0d8c65cf-f75b-40d5-85e4-f07d93b82995"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:44.647708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.647626    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m44h\" (UniqueName: \"kubernetes.io/projected/0d8c65cf-f75b-40d5-85e4-f07d93b82995-kube-api-access-2m44h\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:44.647708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:44.647659    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8c65cf-f75b-40d5-85e4-f07d93b82995-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:45.191559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:45.191529    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"
Oct 30 19:30:45.191559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:45.191540    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7" event={"ID":"0d8c65cf-f75b-40d5-85e4-f07d93b82995","Type":"ContainerDied","Data":"a19e4081380e2d1f6a4b0916204f9d2af9bc3d80d9a8f9869ed1d1ec157369be"}
Oct 30 19:30:45.192004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:45.191588    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a19e4081380e2d1f6a4b0916204f9d2af9bc3d80d9a8f9869ed1d1ec157369be"
Oct 30 19:30:50.216414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.216377    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"]
Oct 30 19:30:50.217246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217229    6777 cpu_manager.go:414] "containerMap: removing container" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy"
Oct 30 19:30:50.217310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217246    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy"
Oct 30 19:30:50.217310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217282    6777 cpu_manager.go:414] "containerMap: removing container" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy"
Oct 30 19:30:50.217310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217288    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy"
Oct 30 19:30:50.217310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217296    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy"
Oct 30 19:30:50.217310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217301    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217313    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217320    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217332    6777 cpu_manager.go:414] "containerMap: removing container" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerName="cleanup"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217336    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfdcc5a-3ef1-418d-b5d6-ffdfcf8ee23c" containerName="cleanup"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217342    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0d8c65cf-f75b-40d5-85e4-f07d93b82995" containerName="revoke-keys"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217346    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8c65cf-f75b-40d5-85e4-f07d93b82995" containerName="revoke-keys"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217357    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" containerName="cleanup"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217362    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" containerName="cleanup"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217370    6777 cpu_manager.go:414] "containerMap: removing container" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy"
Oct 30 19:30:50.217467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217374    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy"
Oct 30 19:30:50.217746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217478    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="4959594f-0d2e-49ed-a21a-17db03609138" containerName="istio-proxy"
Oct 30 19:30:50.217746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217489    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56ca541-7f88-46cd-aa63-beb3988e72cd" containerName="istio-proxy"
Oct 30 19:30:50.217746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217496    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="173b9bfc-03f9-4604-b5b5-3f2d0625d231" containerName="cleanup"
Oct 30 19:30:50.217746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217502    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6449741c-acf1-41b6-80dd-e3416a04a548" containerName="istio-proxy"
Oct 30 19:30:50.217746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217512    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8c65cf-f75b-40d5-85e4-f07d93b82995" containerName="revoke-keys"
Oct 30 19:30:50.217746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217519    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b593d1-d5dc-454e-8ad5-a9d6ad57b8b3" containerName="istio-proxy"
Oct 30 19:30:50.217746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217526    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1276cd7c-e245-4663-a46f-d03854c463c7" containerName="cleanup"
Oct 30 19:30:50.217746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.217535    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f1534e-4438-4789-aeee-f944b3dd15fc" containerName="istio-proxy"
Oct 30 19:30:50.221333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.221310    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.232344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.230778    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:30:50.232344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.230887    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:30:50.232344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.231953    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"]
Oct 30 19:30:50.302111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.299469    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.302111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.299598    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.302111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.299775    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwqs\" (UniqueName: \"kubernetes.io/projected/a7232cee-b66b-49fb-b789-e7a814bfc2a7-kube-api-access-tkwqs\") pod \"maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.416080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.410589    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.416080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.410681    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.416080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.410890    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwqs\" (UniqueName: \"kubernetes.io/projected/a7232cee-b66b-49fb-b789-e7a814bfc2a7-kube-api-access-tkwqs\") pod \"maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.416080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.412746    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.428082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.426233    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.435999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.435241    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwqs\" (UniqueName: \"kubernetes.io/projected/a7232cee-b66b-49fb-b789-e7a814bfc2a7-kube-api-access-tkwqs\") pod \"maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.554112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.550552    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:50.878442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:50.858709    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"]
Oct 30 19:30:50.878442 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:30:50.876180    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7232cee_b66b_49fb_b789_e7a814bfc2a7.slice/crio-b71357c1b435619ebdc13c9b552b9d3a34ee1e4cb7121b506454117785309e30 WatchSource:0}: Error finding container b71357c1b435619ebdc13c9b552b9d3a34ee1e4cb7121b506454117785309e30: Status 404 returned error can't find the container with id b71357c1b435619ebdc13c9b552b9d3a34ee1e4cb7121b506454117785309e30
Oct 30 19:30:51.057598 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.040967    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"]
Oct 30 19:30:51.057598 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.041226    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy" containerID="cri-o://af2f36699131ef9ef25942df56ef8e2e3dd969fb36c522aeff0ace152104d833" gracePeriod=30
Oct 30 19:30:51.180615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.176037    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"]
Oct 30 19:30:51.180615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.176381    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy" containerID="cri-o://8f09d5ca7a902aedc7a8a9728a33624d0cf698b94b2cc39db08e7813a4cde566" gracePeriod=30
Oct 30 19:30:51.180615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.177537    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"]
Oct 30 19:30:51.180615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.177818    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy" containerID="cri-o://9deb101b0be2e1c6072648462dc016eb8be082ee27fb9a6cd618f8d29fb31b81" gracePeriod=30
Oct 30 19:30:51.199495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.198043    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"]
Oct 30 19:30:51.258912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.258871    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc" event={"ID":"a7232cee-b66b-49fb-b789-e7a814bfc2a7","Type":"ContainerStarted","Data":"b71357c1b435619ebdc13c9b552b9d3a34ee1e4cb7121b506454117785309e30"}
Oct 30 19:30:51.259373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.259157    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy" containerID="cri-o://f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66" gracePeriod=30
Oct 30 19:30:51.451567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.451490    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"]
Oct 30 19:30:51.460893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:51.460859    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"]
Oct 30 19:30:52.297705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:52.297670    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc" event={"ID":"a7232cee-b66b-49fb-b789-e7a814bfc2a7","Type":"ContainerStarted","Data":"347fbe097f3fb3a6717a3a117bb98938c8d9696ce03ef638d5c7b316dc017f8b"}
Oct 30 19:30:53.307999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.307964    6777 generic.go:356] "Generic (PLEG): container finished" podID="a7232cee-b66b-49fb-b789-e7a814bfc2a7" containerID="347fbe097f3fb3a6717a3a117bb98938c8d9696ce03ef638d5c7b316dc017f8b" exitCode=0
Oct 30 19:30:53.308501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.308097    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc" event={"ID":"a7232cee-b66b-49fb-b789-e7a814bfc2a7","Type":"ContainerDied","Data":"347fbe097f3fb3a6717a3a117bb98938c8d9696ce03ef638d5c7b316dc017f8b"}
Oct 30 19:30:53.520899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.520871    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:53.694126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.687678    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-xks-ca\") pod \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") "
Oct 30 19:30:53.694126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.687806    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-openshift-ca\") pod \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") "
Oct 30 19:30:53.694126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.688030    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7232cee-b66b-49fb-b789-e7a814bfc2a7-kube-api-access-tkwqs\" (UniqueName: \"kubernetes.io/projected/a7232cee-b66b-49fb-b789-e7a814bfc2a7-kube-api-access-tkwqs\") pod \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\" (UID: \"a7232cee-b66b-49fb-b789-e7a814bfc2a7\") "
Oct 30 19:30:53.708087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.704300    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-openshift-ca" pod "a7232cee-b66b-49fb-b789-e7a814bfc2a7" (UID: "a7232cee-b66b-49fb-b789-e7a814bfc2a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:53.742746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.728120    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7232cee-b66b-49fb-b789-e7a814bfc2a7-kube-api-access-tkwqs" pod "a7232cee-b66b-49fb-b789-e7a814bfc2a7" (UID: "a7232cee-b66b-49fb-b789-e7a814bfc2a7"). InnerVolumeSpecName "kube-api-access-tkwqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:53.742746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.728556    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-xks-ca" pod "a7232cee-b66b-49fb-b789-e7a814bfc2a7" (UID: "a7232cee-b66b-49fb-b789-e7a814bfc2a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:53.790492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.790430    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkwqs\" (UniqueName: \"kubernetes.io/projected/a7232cee-b66b-49fb-b789-e7a814bfc2a7-kube-api-access-tkwqs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:53.790492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.790462    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:53.790492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:53.790471    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7232cee-b66b-49fb-b789-e7a814bfc2a7-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:54.318475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.318427    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc" event={"ID":"a7232cee-b66b-49fb-b789-e7a814bfc2a7","Type":"ContainerDied","Data":"b71357c1b435619ebdc13c9b552b9d3a34ee1e4cb7121b506454117785309e30"}
Oct 30 19:30:54.318475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.318457    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"
Oct 30 19:30:54.318888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.318505    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b71357c1b435619ebdc13c9b552b9d3a34ee1e4cb7121b506454117785309e30"
Oct 30 19:30:54.782412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.782356    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"]
Oct 30 19:30:54.783283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.783267    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a7232cee-b66b-49fb-b789-e7a814bfc2a7" containerName="revoke-keys"
Oct 30 19:30:54.783337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.783284    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7232cee-b66b-49fb-b789-e7a814bfc2a7" containerName="revoke-keys"
Oct 30 19:30:54.783484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.783471    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7232cee-b66b-49fb-b789-e7a814bfc2a7" containerName="revoke-keys"
Oct 30 19:30:54.787838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.787748    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:54.793785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.793757    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-891d80\"/\"test-model-891d80-kserve-self-signed-certs\""
Oct 30 19:30:54.793939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.793817    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-891d80\"/\"kube-root-ca.crt\""
Oct 30 19:30:54.793939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.793857    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-891d80\"/\"default-dockercfg-rp797\""
Oct 30 19:30:54.793939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.793879    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-891d80\"/\"openshift-service-ca.crt\""
Oct 30 19:30:54.794160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.794143    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"]
Oct 30 19:30:54.909840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.909809    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-home\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:54.910032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.909879    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-model-cache\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:54.910032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.909989    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tmp-dir\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:54.910245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.910229    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xv8\" (UniqueName: \"kubernetes.io/projected/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-kube-api-access-p9xv8\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:54.910297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.910283    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-dshm\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:54.910448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:54.910427    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tls-certs\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.026081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.022434    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xv8\" (UniqueName: \"kubernetes.io/projected/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-kube-api-access-p9xv8\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.026081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.022530    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-dshm\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.026081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.023500    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tls-certs\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.026081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.023582    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-home\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.026081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.023660    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-model-cache\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.026081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.025961    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tmp-dir\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.026522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.025870    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-home\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.026522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.024449    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-model-cache\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.026522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.026364    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tmp-dir\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.033899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.033828    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tls-certs\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.073293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.072404    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-dshm\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.096047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.095870    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xv8\" (UniqueName: \"kubernetes.io/projected/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-kube-api-access-p9xv8\") pod \"test-model-891d80-kserve-8677659fff-49fzb\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") " pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.126765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.126716    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:55.512083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:55.511763    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"]
Oct 30 19:30:56.346653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.346612    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb" event={"ID":"cfd52fec-93ab-4d12-a8d7-38fdae88fa92","Type":"ContainerStarted","Data":"19397e3b411fc6f628ff11789df9df0f334fd268e51b51ab360cea145a3638f4"}
Oct 30 19:30:56.346798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.346666    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb" event={"ID":"cfd52fec-93ab-4d12-a8d7-38fdae88fa92","Type":"ContainerStarted","Data":"c6734b31db78dde4cf4cff0c945c3400d4488f7ef070e5136fd5db15a37df6b5"}
Oct 30 19:30:56.363119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.362142    6777 generic.go:356] "Generic (PLEG): container finished" podID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerID="af2f36699131ef9ef25942df56ef8e2e3dd969fb36c522aeff0ace152104d833" exitCode=0
Oct 30 19:30:56.363119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.362275    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" event={"ID":"5c80e7ec-fe29-4139-a608-d10d0fb4901d","Type":"ContainerDied","Data":"af2f36699131ef9ef25942df56ef8e2e3dd969fb36c522aeff0ace152104d833"}
Oct 30 19:30:56.388117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.385785    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb" podStartSLOduration=2.385765115 podStartE2EDuration="2.385765115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:56.374597616 +0000 UTC m=+1438.371513001" watchObservedRunningTime="2026-10-30 19:30:56.385765115 +0000 UTC m=+1438.382680500"
Oct 30 19:30:56.388846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.388814    6777 generic.go:356] "Generic (PLEG): container finished" podID="2eac6694-e9ee-4636-8521-dca300646ba2" containerID="9deb101b0be2e1c6072648462dc016eb8be082ee27fb9a6cd618f8d29fb31b81" exitCode=0
Oct 30 19:30:56.389045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.389025    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" event={"ID":"2eac6694-e9ee-4636-8521-dca300646ba2","Type":"ContainerDied","Data":"9deb101b0be2e1c6072648462dc016eb8be082ee27fb9a6cd618f8d29fb31b81"}
Oct 30 19:30:56.391313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.391289    6777 generic.go:356] "Generic (PLEG): container finished" podID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerID="8f09d5ca7a902aedc7a8a9728a33624d0cf698b94b2cc39db08e7813a4cde566" exitCode=0
Oct 30 19:30:56.391430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.391416    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" event={"ID":"89a618ff-60dc-4c04-aaca-1f40b343a57e","Type":"ContainerDied","Data":"8f09d5ca7a902aedc7a8a9728a33624d0cf698b94b2cc39db08e7813a4cde566"}
Oct 30 19:30:56.469142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.452128    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m" podUID="ff6f67de-1e81-4bc3-9489-d2a24bbba61a" containerName="payload-processing" containerID="cri-o://4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372" gracePeriod=25
Oct 30 19:30:56.469142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.466797    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w" podUID="8059c66f-eff9-4f08-99a3-9c216aa17183" containerName="payload-pre-processing" containerID="cri-o://9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9" gracePeriod=25
Oct 30 19:30:56.640048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.640022    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:30:56.740642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.740520    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-envoy\") pod \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") "
Oct 30 19:30:56.740642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.740607    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-credential-socket\") pod \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") "
Oct 30 19:30:56.740642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.740652    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-token\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-token\") pod \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") "
Oct 30 19:30:56.740916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.740854    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istiod-ca-cert\") pod \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") "
Oct 30 19:30:56.741036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.740924    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-podinfo\") pod \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") "
Oct 30 19:30:56.741036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.740980    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-certs\") pod \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") "
Oct 30 19:30:56.741036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.741015    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-kube-api-access-nwfqj\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-kube-api-access-nwfqj\") pod \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") "
Oct 30 19:30:56.741947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.741206    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-socket\") pod \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") "
Oct 30 19:30:56.741947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.741274    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-data\") pod \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\" (UID: \"5c80e7ec-fe29-4139-a608-d10d0fb4901d\") "
Oct 30 19:30:56.741947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.741475    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istiod-ca-cert" pod "5c80e7ec-fe29-4139-a608-d10d0fb4901d" (UID: "5c80e7ec-fe29-4139-a608-d10d0fb4901d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:56.741947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.741745    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-credential-socket" pod "5c80e7ec-fe29-4139-a608-d10d0fb4901d" (UID: "5c80e7ec-fe29-4139-a608-d10d0fb4901d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.742530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.742305    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.742530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.742321    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-data" pod "5c80e7ec-fe29-4139-a608-d10d0fb4901d" (UID: "5c80e7ec-fe29-4139-a608-d10d0fb4901d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.742530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.742332    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.742739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.742536    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-socket" pod "5c80e7ec-fe29-4139-a608-d10d0fb4901d" (UID: "5c80e7ec-fe29-4139-a608-d10d0fb4901d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.743863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.743787    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-certs" pod "5c80e7ec-fe29-4139-a608-d10d0fb4901d" (UID: "5c80e7ec-fe29-4139-a608-d10d0fb4901d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.747266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.746485    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-envoy" pod "5c80e7ec-fe29-4139-a608-d10d0fb4901d" (UID: "5c80e7ec-fe29-4139-a608-d10d0fb4901d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.747266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.746600    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-podinfo" pod "5c80e7ec-fe29-4139-a608-d10d0fb4901d" (UID: "5c80e7ec-fe29-4139-a608-d10d0fb4901d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:56.747266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.746971    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-token" pod "5c80e7ec-fe29-4139-a608-d10d0fb4901d" (UID: "5c80e7ec-fe29-4139-a608-d10d0fb4901d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:56.797277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.797236    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-kube-api-access-nwfqj" pod "5c80e7ec-fe29-4139-a608-d10d0fb4901d" (UID: "5c80e7ec-fe29-4139-a608-d10d0fb4901d"). InnerVolumeSpecName "kube-api-access-nwfqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:56.807197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.806770    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:30:56.846555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.846522    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:30:56.847830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.847799    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-token\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-token\") pod \"89a618ff-60dc-4c04-aaca-1f40b343a57e\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") "
Oct 30 19:30:56.860107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.857748    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-certs" pod "89a618ff-60dc-4c04-aaca-1f40b343a57e" (UID: "89a618ff-60dc-4c04-aaca-1f40b343a57e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.880844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.847897    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-certs\") pod \"89a618ff-60dc-4c04-aaca-1f40b343a57e\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") "
Oct 30 19:30:56.882184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.882151    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a618ff-60dc-4c04-aaca-1f40b343a57e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a618ff-60dc-4c04-aaca-1f40b343a57e-istiod-ca-cert\") pod \"89a618ff-60dc-4c04-aaca-1f40b343a57e\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") "
Oct 30 19:30:56.882317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.882261    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-data\") pod \"89a618ff-60dc-4c04-aaca-1f40b343a57e\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") "
Oct 30 19:30:56.882680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.882655    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a618ff-60dc-4c04-aaca-1f40b343a57e-istiod-ca-cert" pod "89a618ff-60dc-4c04-aaca-1f40b343a57e" (UID: "89a618ff-60dc-4c04-aaca-1f40b343a57e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:56.882937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.882904    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-podinfo\") pod \"89a618ff-60dc-4c04-aaca-1f40b343a57e\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") "
Oct 30 19:30:56.883040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.882958    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-credential-socket\") pod \"89a618ff-60dc-4c04-aaca-1f40b343a57e\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") "
Oct 30 19:30:56.883040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.883002    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-socket\") pod \"89a618ff-60dc-4c04-aaca-1f40b343a57e\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") "
Oct 30 19:30:56.883205 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.883181    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-envoy\") pod \"89a618ff-60dc-4c04-aaca-1f40b343a57e\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") "
Oct 30 19:30:56.883275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.883240    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-kube-api-access-mkvp5\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-kube-api-access-mkvp5\") pod \"89a618ff-60dc-4c04-aaca-1f40b343a57e\" (UID: \"89a618ff-60dc-4c04-aaca-1f40b343a57e\") "
Oct 30 19:30:56.884381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.884358    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-socket" pod "89a618ff-60dc-4c04-aaca-1f40b343a57e" (UID: "89a618ff-60dc-4c04-aaca-1f40b343a57e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.884685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.884670    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-data" pod "89a618ff-60dc-4c04-aaca-1f40b343a57e" (UID: "89a618ff-60dc-4c04-aaca-1f40b343a57e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885040    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885086    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885100    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885115    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885131    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885130    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-credential-socket" pod "89a618ff-60dc-4c04-aaca-1f40b343a57e" (UID: "89a618ff-60dc-4c04-aaca-1f40b343a57e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885143    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a618ff-60dc-4c04-aaca-1f40b343a57e-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885166    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885182    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80e7ec-fe29-4139-a608-d10d0fb4901d-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885198    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80e7ec-fe29-4139-a608-d10d0fb4901d-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885212    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwfqj\" (UniqueName: \"kubernetes.io/projected/5c80e7ec-fe29-4139-a608-d10d0fb4901d-kube-api-access-nwfqj\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.885248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.885226    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.899255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.899200    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-envoy" pod "89a618ff-60dc-4c04-aaca-1f40b343a57e" (UID: "89a618ff-60dc-4c04-aaca-1f40b343a57e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.907450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.904451    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-token" pod "89a618ff-60dc-4c04-aaca-1f40b343a57e" (UID: "89a618ff-60dc-4c04-aaca-1f40b343a57e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:56.907450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.904543    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-kube-api-access-mkvp5" pod "89a618ff-60dc-4c04-aaca-1f40b343a57e" (UID: "89a618ff-60dc-4c04-aaca-1f40b343a57e"). InnerVolumeSpecName "kube-api-access-mkvp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:56.907450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.907211    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-podinfo" pod "89a618ff-60dc-4c04-aaca-1f40b343a57e" (UID: "89a618ff-60dc-4c04-aaca-1f40b343a57e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:56.999174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.999143    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-envoy\") pod \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") "
Oct 30 19:30:56.999416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:56.999399    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-podinfo\") pod \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") "
Oct 30 19:30:57.013026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.012945    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-podinfo" pod "44c4a8a2-fc32-4c4c-ae35-27d7afda27db" (UID: "44c4a8a2-fc32-4c4c-ae35-27d7afda27db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:57.015105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.014750    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-envoy" pod "44c4a8a2-fc32-4c4c-ae35-27d7afda27db" (UID: "44c4a8a2-fc32-4c4c-ae35-27d7afda27db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.015330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.015306    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istiod-ca-cert\") pod \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") "
Oct 30 19:30:57.015539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.015524    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-token\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-token\") pod \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") "
Oct 30 19:30:57.015685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.015672    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-certs\") pod \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") "
Oct 30 19:30:57.015813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.015799    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-socket\") pod \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") "
Oct 30 19:30:57.015977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.015964    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-data\") pod \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") "
Oct 30 19:30:57.016713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.016693    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-credential-socket\") pod \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") "
Oct 30 19:30:57.017046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.017025    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-kube-api-access-b4gvc\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-kube-api-access-b4gvc\") pod \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\" (UID: \"44c4a8a2-fc32-4c4c-ae35-27d7afda27db\") "
Oct 30 19:30:57.018533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.018513    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.018646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.018632    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkvp5\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-kube-api-access-mkvp5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.018723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.018712    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.018812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.018801    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.018894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.018883    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.018972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.018954    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a618ff-60dc-4c04-aaca-1f40b343a57e-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.019089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.019077    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a618ff-60dc-4c04-aaca-1f40b343a57e-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.030464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.030435    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-certs" pod "44c4a8a2-fc32-4c4c-ae35-27d7afda27db" (UID: "44c4a8a2-fc32-4c4c-ae35-27d7afda27db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.030793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.030775    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-socket" pod "44c4a8a2-fc32-4c4c-ae35-27d7afda27db" (UID: "44c4a8a2-fc32-4c4c-ae35-27d7afda27db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.031220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.031200    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-data" pod "44c4a8a2-fc32-4c4c-ae35-27d7afda27db" (UID: "44c4a8a2-fc32-4c4c-ae35-27d7afda27db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.031540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.031512    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-credential-socket" pod "44c4a8a2-fc32-4c4c-ae35-27d7afda27db" (UID: "44c4a8a2-fc32-4c4c-ae35-27d7afda27db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.031654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.031549    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istiod-ca-cert" pod "44c4a8a2-fc32-4c4c-ae35-27d7afda27db" (UID: "44c4a8a2-fc32-4c4c-ae35-27d7afda27db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:57.031733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.031671    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-kube-api-access-b4gvc" pod "44c4a8a2-fc32-4c4c-ae35-27d7afda27db" (UID: "44c4a8a2-fc32-4c4c-ae35-27d7afda27db"). InnerVolumeSpecName "kube-api-access-b4gvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:57.044938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.044905    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"]
Oct 30 19:30:57.087234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.047792    6777 cpu_manager.go:414] "containerMap: removing container" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy"
Oct 30 19:30:57.087234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.047818    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy"
Oct 30 19:30:57.087234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.047873    6777 cpu_manager.go:414] "containerMap: removing container" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy"
Oct 30 19:30:57.087234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.047882    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy"
Oct 30 19:30:57.087234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.047897    6777 cpu_manager.go:414] "containerMap: removing container" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy"
Oct 30 19:30:57.087234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.047905    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy"
Oct 30 19:30:57.087234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.048450    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" containerName="istio-proxy"
Oct 30 19:30:57.087234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.048476    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" containerName="istio-proxy"
Oct 30 19:30:57.087234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.048488    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerName="istio-proxy"
Oct 30 19:30:57.090422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.090313    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.091805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.091637    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-token" pod "44c4a8a2-fc32-4c4c-ae35-27d7afda27db" (UID: "44c4a8a2-fc32-4c4c-ae35-27d7afda27db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:57.099190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.099166    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"]
Oct 30 19:30:57.120110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.120078    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4gvc\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-kube-api-access-b4gvc\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.120323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.120307    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.120417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.120407    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.120497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.120487    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.120580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.120570    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.120650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.120641    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.120718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.120708    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4a8a2-fc32-4c4c-ae35-27d7afda27db-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.130279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.130245    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:30:57.138429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.138397    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-69f3a9-openshift-default-dockercfg-fsxb8\""
Oct 30 19:30:57.269137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.254477    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-envoy\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.269137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.254557    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4wpj\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-kube-api-access-v4wpj\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.269137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.267271    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:30:57.308324 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.308288    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:30:57.331087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.327379    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-data\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.331087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.327558    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-credential-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.331087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.327637    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d65393-28cb-433f-9157-0ad2f5e90083-istiod-ca-cert\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.331087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.327768    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-certs\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.331087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.327804    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-istio-token\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.331087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.327841    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d65393-28cb-433f-9157-0ad2f5e90083-istio-podinfo\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.331087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.327984    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.521699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.521621    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8059c66f-eff9-4f08-99a3-9c216aa17183-kube-api-access-kbxqx\" (UniqueName: \"kubernetes.io/projected/8059c66f-eff9-4f08-99a3-9c216aa17183-kube-api-access-kbxqx\") pod \"8059c66f-eff9-4f08-99a3-9c216aa17183\" (UID: \"8059c66f-eff9-4f08-99a3-9c216aa17183\") "
Oct 30 19:30:57.521835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.521697    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-plugins-config-volume\") pod \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\" (UID: \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\") "
Oct 30 19:30:57.521835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.521767    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8059c66f-eff9-4f08-99a3-9c216aa17183-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8059c66f-eff9-4f08-99a3-9c216aa17183-plugins-config-volume\") pod \"8059c66f-eff9-4f08-99a3-9c216aa17183\" (UID: \"8059c66f-eff9-4f08-99a3-9c216aa17183\") "
Oct 30 19:30:57.522018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522000    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-kube-api-access-bjcl8\" (UniqueName: \"kubernetes.io/projected/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-kube-api-access-bjcl8\") pod \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\" (UID: \"ff6f67de-1e81-4bc3-9489-d2a24bbba61a\") "
Oct 30 19:30:57.522461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522436    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-envoy\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.522566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522491    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4wpj\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-kube-api-access-v4wpj\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.522631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522563    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-data\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.522715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522642    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-credential-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.522715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522686    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d65393-28cb-433f-9157-0ad2f5e90083-istiod-ca-cert\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.522822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522748    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-certs\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.522822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522779    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-istio-token\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.522822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522809    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d65393-28cb-433f-9157-0ad2f5e90083-istio-podinfo\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.522990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.522901    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.525212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.525185    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-data\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.525641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.525613    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-plugins-config-volume" pod "ff6f67de-1e81-4bc3-9489-d2a24bbba61a" (UID: "ff6f67de-1e81-4bc3-9489-d2a24bbba61a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:57.526273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.526244    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-credential-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.526903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.526877    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d65393-28cb-433f-9157-0ad2f5e90083-istiod-ca-cert\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.528279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.527200    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-certs\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.528279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.527365    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8059c66f-eff9-4f08-99a3-9c216aa17183-plugins-config-volume" pod "8059c66f-eff9-4f08-99a3-9c216aa17183" (UID: "8059c66f-eff9-4f08-99a3-9c216aa17183"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:57.528915 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.528510    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8059c66f-eff9-4f08-99a3-9c216aa17183-kube-api-access-kbxqx" pod "8059c66f-eff9-4f08-99a3-9c216aa17183" (UID: "8059c66f-eff9-4f08-99a3-9c216aa17183"). InnerVolumeSpecName "kube-api-access-kbxqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:57.537772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.537515    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-kube-api-access-bjcl8" pod "ff6f67de-1e81-4bc3-9489-d2a24bbba61a" (UID: "ff6f67de-1e81-4bc3-9489-d2a24bbba61a"). InnerVolumeSpecName "kube-api-access-bjcl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:57.537772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.537729    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d65393-28cb-433f-9157-0ad2f5e90083-istio-podinfo\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.538665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.538602    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.540833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.540493    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-envoy\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.547562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.547526    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4wpj\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-kube-api-access-v4wpj\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.572673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.572628    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-istio-token\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:57.583283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.583246    6777 generic.go:356] "Generic (PLEG): container finished" podID="ff6f67de-1e81-4bc3-9489-d2a24bbba61a" containerID="4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372" exitCode=0
Oct 30 19:30:57.583474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.583435    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m" event={"ID":"ff6f67de-1e81-4bc3-9489-d2a24bbba61a","Type":"ContainerDied","Data":"4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372"}
Oct 30 19:30:57.583547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.583478    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m" event={"ID":"ff6f67de-1e81-4bc3-9489-d2a24bbba61a","Type":"ContainerDied","Data":"1480e9903f1f1b8d1f7f5a6e3a1a59858e1b79d2ddf41e33f12b08b48bc74dfe"}
Oct 30 19:30:57.583547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.583509    6777 scope.go:122] "RemoveContainer" containerID="4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372"
Oct 30 19:30:57.583661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.583626    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"
Oct 30 19:30:57.595395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.595353    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5" event={"ID":"5c80e7ec-fe29-4139-a608-d10d0fb4901d","Type":"ContainerDied","Data":"a1ef05303f93a3a2c471a6e167859b261a6a4e9173dd4b46c4a47a9096f99ca9"}
Oct 30 19:30:57.595629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.595556    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"
Oct 30 19:30:57.598348 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.598317    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:30:57.602082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.601208    6777 generic.go:356] "Generic (PLEG): container finished" podID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" containerID="f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66" exitCode=0
Oct 30 19:30:57.602082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.601448    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" event={"ID":"44c4a8a2-fc32-4c4c-ae35-27d7afda27db","Type":"ContainerDied","Data":"f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66"}
Oct 30 19:30:57.602082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.601499    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj" event={"ID":"44c4a8a2-fc32-4c4c-ae35-27d7afda27db","Type":"ContainerDied","Data":"6beabc99ccb08d50300a453e02cf3efa9570ef2020a27addeeb15b53ca6af626"}
Oct 30 19:30:57.602082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.601598    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"
Oct 30 19:30:57.607247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.607215    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw" event={"ID":"89a618ff-60dc-4c04-aaca-1f40b343a57e","Type":"ContainerDied","Data":"3e43ac46ac2a10d2a3b58f504b39a209d056084d9f9e6da2ffd4ec5e245ac4d7"}
Oct 30 19:30:57.607347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.607330    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"
Oct 30 19:30:57.621557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.621531    6777 generic.go:356] "Generic (PLEG): container finished" podID="8059c66f-eff9-4f08-99a3-9c216aa17183" containerID="9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9" exitCode=0
Oct 30 19:30:57.621767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.621748    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"
Oct 30 19:30:57.622601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.621684    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w" event={"ID":"8059c66f-eff9-4f08-99a3-9c216aa17183","Type":"ContainerDied","Data":"9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9"}
Oct 30 19:30:57.622773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.622755    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w" event={"ID":"8059c66f-eff9-4f08-99a3-9c216aa17183","Type":"ContainerDied","Data":"f46074fdae3e0d4094080cc8ee6445b2ac482d5c47dc801a90edaaa061b7fac6"}
Oct 30 19:30:57.625044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.625024    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-socket\") pod \"2eac6694-e9ee-4636-8521-dca300646ba2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") "
Oct 30 19:30:57.625265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.625244    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-data\") pod \"2eac6694-e9ee-4636-8521-dca300646ba2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") "
Oct 30 19:30:57.625409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.625392    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-envoy\") pod \"2eac6694-e9ee-4636-8521-dca300646ba2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") "
Oct 30 19:30:57.625555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.625308    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-socket" pod "2eac6694-e9ee-4636-8521-dca300646ba2" (UID: "2eac6694-e9ee-4636-8521-dca300646ba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.625708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.625518    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-data" pod "2eac6694-e9ee-4636-8521-dca300646ba2" (UID: "2eac6694-e9ee-4636-8521-dca300646ba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.625825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.625565    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-istio-token\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-istio-token\") pod \"2eac6694-e9ee-4636-8521-dca300646ba2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") "
Oct 30 19:30:57.626003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.625978    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eac6694-e9ee-4636-8521-dca300646ba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eac6694-e9ee-4636-8521-dca300646ba2-istiod-ca-cert\") pod \"2eac6694-e9ee-4636-8521-dca300646ba2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") "
Oct 30 19:30:57.626090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.626046    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-kube-api-access-trb84\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-kube-api-access-trb84\") pod \"2eac6694-e9ee-4636-8521-dca300646ba2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") "
Oct 30 19:30:57.626156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.626126    6777 scope.go:122] "RemoveContainer" containerID="4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372"
Oct 30 19:30:57.626211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.626183    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-certs\") pod \"2eac6694-e9ee-4636-8521-dca300646ba2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") "
Oct 30 19:30:57.626320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.626229    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eac6694-e9ee-4636-8521-dca300646ba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eac6694-e9ee-4636-8521-dca300646ba2-istio-podinfo\") pod \"2eac6694-e9ee-4636-8521-dca300646ba2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") "
Oct 30 19:30:57.626387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.626359    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-credential-socket\") pod \"2eac6694-e9ee-4636-8521-dca300646ba2\" (UID: \"2eac6694-e9ee-4636-8521-dca300646ba2\") "
Oct 30 19:30:57.627450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.627264    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.627450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.627325    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.627450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.627341    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbxqx\" (UniqueName: \"kubernetes.io/projected/8059c66f-eff9-4f08-99a3-9c216aa17183-kube-api-access-kbxqx\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.627450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.627357    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.627450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.627371    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8059c66f-eff9-4f08-99a3-9c216aa17183-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.627450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.627386    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjcl8\" (UniqueName: \"kubernetes.io/projected/ff6f67de-1e81-4bc3-9489-d2a24bbba61a-kube-api-access-bjcl8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.627450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.627271    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eac6694-e9ee-4636-8521-dca300646ba2-istiod-ca-cert" pod "2eac6694-e9ee-4636-8521-dca300646ba2" (UID: "2eac6694-e9ee-4636-8521-dca300646ba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:57.627450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.627428    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-certs" pod "2eac6694-e9ee-4636-8521-dca300646ba2" (UID: "2eac6694-e9ee-4636-8521-dca300646ba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.628108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.627577    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-credential-socket" pod "2eac6694-e9ee-4636-8521-dca300646ba2" (UID: "2eac6694-e9ee-4636-8521-dca300646ba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.628496 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:57.627354    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372\": container with ID starting with 4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372 not found: ID does not exist" containerID="4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372"
Oct 30 19:30:57.628496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.628318    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372"} err="failed to get container status \"4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372\": rpc error: code = NotFound desc = could not find container \"4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372\": container with ID starting with 4aade68463bbd34bfa62ab95d330d799675f43bb2e63066a5dc93722a0895372 not found: ID does not exist"
Oct 30 19:30:57.628496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.628348    6777 scope.go:122] "RemoveContainer" containerID="af2f36699131ef9ef25942df56ef8e2e3dd969fb36c522aeff0ace152104d833"
Oct 30 19:30:57.629916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.629878    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-istio-token" pod "2eac6694-e9ee-4636-8521-dca300646ba2" (UID: "2eac6694-e9ee-4636-8521-dca300646ba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:57.630588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.630568    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-envoy" pod "2eac6694-e9ee-4636-8521-dca300646ba2" (UID: "2eac6694-e9ee-4636-8521-dca300646ba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:57.631463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.631438    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-kube-api-access-trb84" pod "2eac6694-e9ee-4636-8521-dca300646ba2" (UID: "2eac6694-e9ee-4636-8521-dca300646ba2"). InnerVolumeSpecName "kube-api-access-trb84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:57.642008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.641977    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eac6694-e9ee-4636-8521-dca300646ba2-istio-podinfo" pod "2eac6694-e9ee-4636-8521-dca300646ba2" (UID: "2eac6694-e9ee-4636-8521-dca300646ba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:57.689449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.689412    6777 scope.go:122] "RemoveContainer" containerID="f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66"
Oct 30 19:30:57.693647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.693625    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"]
Oct 30 19:30:57.697717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.697693    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-xxhlw"]
Oct 30 19:30:57.703235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.703215    6777 scope.go:122] "RemoveContainer" containerID="f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66"
Oct 30 19:30:57.703618 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:57.703576    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66\": container with ID starting with f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66 not found: ID does not exist" containerID="f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66"
Oct 30 19:30:57.703780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.703753    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66"} err="failed to get container status \"f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66\": rpc error: code = NotFound desc = could not find container \"f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66\": container with ID starting with f93a0a3a9e5a71f815dd0e729d5f1aa6e8695dcc19810710913efcb4ba50cf66 not found: ID does not exist"
Oct 30 19:30:57.703874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.703863    6777 scope.go:122] "RemoveContainer" containerID="8f09d5ca7a902aedc7a8a9728a33624d0cf698b94b2cc39db08e7813a4cde566"
Oct 30 19:30:57.707771 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.707745    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"]
Oct 30 19:30:57.712665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.712643    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4012a-b58495cbc-zdz5w"]
Oct 30 19:30:57.718452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.718427    6777 scope.go:122] "RemoveContainer" containerID="9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9"
Oct 30 19:30:57.727863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.727821    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"]
Oct 30 19:30:57.729538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.729444    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trb84\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-kube-api-access-trb84\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.729538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.729464    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.729538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.729479    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eac6694-e9ee-4636-8521-dca300646ba2-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.729538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.729490    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.729538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.729498    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eac6694-e9ee-4636-8521-dca300646ba2-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.729538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.729506    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eac6694-e9ee-4636-8521-dca300646ba2-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.729538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.729515    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eac6694-e9ee-4636-8521-dca300646ba2-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:57.732336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.732319    6777 scope.go:122] "RemoveContainer" containerID="9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9"
Oct 30 19:30:57.732643 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:30:57.732621    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9\": container with ID starting with 9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9 not found: ID does not exist" containerID="9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9"
Oct 30 19:30:57.732718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.732650    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9"} err="failed to get container status \"9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9\": rpc error: code = NotFound desc = could not find container \"9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9\": container with ID starting with 9221424f580e30c509e69e502e69aa2c2554bb7bb153db1860a7e3af146c10f9 not found: ID does not exist"
Oct 30 19:30:57.733270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.733252    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4012a-78c8dd4c94-sgr5m"]
Oct 30 19:30:57.783731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.783685    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"]
Oct 30 19:30:57.787494 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.787466    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-slpf5"]
Oct 30 19:30:57.796347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.796322    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"]
Oct 30 19:30:57.814345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.814315    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-qvtfj"]
Oct 30 19:30:57.844469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:57.844443    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:58.118155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.118122    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c4a8a2-fc32-4c4c-ae35-27d7afda27db" path="/var/lib/kubelet/pods/44c4a8a2-fc32-4c4c-ae35-27d7afda27db/volumes"
Oct 30 19:30:58.118805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.118783    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c80e7ec-fe29-4139-a608-d10d0fb4901d" path="/var/lib/kubelet/pods/5c80e7ec-fe29-4139-a608-d10d0fb4901d/volumes"
Oct 30 19:30:58.119360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.119340    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8059c66f-eff9-4f08-99a3-9c216aa17183" path="/var/lib/kubelet/pods/8059c66f-eff9-4f08-99a3-9c216aa17183/volumes"
Oct 30 19:30:58.121878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.121859    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a618ff-60dc-4c04-aaca-1f40b343a57e" path="/var/lib/kubelet/pods/89a618ff-60dc-4c04-aaca-1f40b343a57e/volumes"
Oct 30 19:30:58.122446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.122424    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6f67de-1e81-4bc3-9489-d2a24bbba61a" path="/var/lib/kubelet/pods/ff6f67de-1e81-4bc3-9489-d2a24bbba61a/volumes"
Oct 30 19:30:58.151940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.151910    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"]
Oct 30 19:30:58.155754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.155712    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:30:58.155856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.155806    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:30:58.155916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.155852    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:30:58.634980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.634935    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" event={"ID":"11d65393-28cb-433f-9157-0ad2f5e90083","Type":"ContainerStarted","Data":"8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496"}
Oct 30 19:30:58.635187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.635019    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" event={"ID":"11d65393-28cb-433f-9157-0ad2f5e90083","Type":"ContainerStarted","Data":"ed3c4d03f47dce9db1605a3c47d07fd016f2ae05819d698023c261c7602dbab3"}
Oct 30 19:30:58.641637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.641609    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"
Oct 30 19:30:58.641637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.641636    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2" event={"ID":"2eac6694-e9ee-4636-8521-dca300646ba2","Type":"ContainerDied","Data":"d9023768d82676110b4f8265d7ea4e09c90cfc9062344f7bac5271ee836f7919"}
Oct 30 19:30:58.641637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.641708    6777 scope.go:122] "RemoveContainer" containerID="9deb101b0be2e1c6072648462dc016eb8be082ee27fb9a6cd618f8d29fb31b81"
Oct 30 19:30:58.664894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.664838    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" podStartSLOduration=1.6648173339999999 podStartE2EDuration="1.664817334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:58.660555889 +0000 UTC m=+1440.657471271" watchObservedRunningTime="2026-10-30 19:30:58.664817334 +0000 UTC m=+1440.661732718"
Oct 30 19:30:58.691572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.691529    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"]
Oct 30 19:30:58.691736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.691595    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-8qnr2"]
Oct 30 19:30:58.845509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:58.845472    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:59.249401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.249367    6777 scope.go:122] "RemoveContainer" containerID="899b729e552acbbc8f1e9fc7fe3f6324163d92a9aba3306fbb0d8ef2290ebd6d"
Oct 30 19:30:59.262222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.262193    6777 scope.go:122] "RemoveContainer" containerID="a6053eff2e797b6b36b1f3aa6a4066bcb40c32b2046426b4926d10e81dd90c09"
Oct 30 19:30:59.274468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.274444    6777 scope.go:122] "RemoveContainer" containerID="ff755b7f47975f312b2c85cbb8209a19da23eab497814a6dc57869977ac7008f"
Oct 30 19:30:59.287953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.287929    6777 scope.go:122] "RemoveContainer" containerID="2f831fc502aacdce1e9681f9f5881052a1babfe4be76c6d29a355153de34e5fc"
Oct 30 19:30:59.301612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.301586    6777 scope.go:122] "RemoveContainer" containerID="0f9c6777623b571847ae79383998ce8ceec1f40e61edf6c68cf2365293973025"
Oct 30 19:30:59.312112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.312090    6777 scope.go:122] "RemoveContainer" containerID="f932186080a30b4b86f62ea8e3d7e265dc2307022ebcf8cf6136bee53df42df1"
Oct 30 19:30:59.322366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.322342    6777 scope.go:122] "RemoveContainer" containerID="09a672bc9cb22dcd480d1b2774e14bcc434e777eb52f64a0c876bd3b9c905b3f"
Oct 30 19:30:59.332739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.332716    6777 scope.go:122] "RemoveContainer" containerID="02205bfe7e9c96c0eeb3eb75ad61ac776e3ca2f5a68b94a71b4680454c003a9b"
Oct 30 19:30:59.350541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.350518    6777 scope.go:122] "RemoveContainer" containerID="7dc9699f22f92746c824aabce8e11c86e030f6fa9b6f0f255dc4b01354bc9619"
Oct 30 19:30:59.716623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.716588    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"]
Oct 30 19:30:59.717603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.717581    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8059c66f-eff9-4f08-99a3-9c216aa17183" containerName="payload-pre-processing"
Oct 30 19:30:59.717723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.717604    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8059c66f-eff9-4f08-99a3-9c216aa17183" containerName="payload-pre-processing"
Oct 30 19:30:59.717723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.717692    6777 cpu_manager.go:414] "containerMap: removing container" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy"
Oct 30 19:30:59.717723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.717701    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy"
Oct 30 19:30:59.717723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.717713    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ff6f67de-1e81-4bc3-9489-d2a24bbba61a" containerName="payload-processing"
Oct 30 19:30:59.717723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.717721    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6f67de-1e81-4bc3-9489-d2a24bbba61a" containerName="payload-processing"
Oct 30 19:30:59.717968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.717901    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8059c66f-eff9-4f08-99a3-9c216aa17183" containerName="payload-pre-processing"
Oct 30 19:30:59.717968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.717930    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" containerName="istio-proxy"
Oct 30 19:30:59.717968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.717941    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6f67de-1e81-4bc3-9489-d2a24bbba61a" containerName="payload-processing"
Oct 30 19:30:59.725853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.725829    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.733295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.733270    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-33b4d7-openshift-default-dockercfg-2x2w9\""
Oct 30 19:30:59.734388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.734360    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"]
Oct 30 19:30:59.785040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.785007    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-token\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.785261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.785232    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-certs\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.785327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.785308    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-podinfo\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.785450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.785434    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istiod-ca-cert\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.785489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.785469    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-credential-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.785655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.785637    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.785776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.785759    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-data\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.785833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.785790    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-envoy\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.785889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.785869    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j77wt\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-kube-api-access-j77wt\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.851888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.851857    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:30:59.886908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.886871    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-podinfo\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.886957    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istiod-ca-cert\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.886992    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-credential-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887037    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887137    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-data\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887171    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-envoy\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887228    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77wt\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-kube-api-access-j77wt\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887396    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-token\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887455    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-credential-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887499    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-data\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887551    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887567    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-certs\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887815    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-certs\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.887897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.887860    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istiod-ca-cert\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.889681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.889659    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-envoy\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.889829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.889811    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-podinfo\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.902977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.902942    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-token\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:30:59.903393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:30:59.903371    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j77wt\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-kube-api-access-j77wt\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:31:00.048323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.048240    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:31:00.119832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.119802    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eac6694-e9ee-4636-8521-dca300646ba2" path="/var/lib/kubelet/pods/2eac6694-e9ee-4636-8521-dca300646ba2/volumes"
Oct 30 19:31:00.352235 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:00.352192    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode498f782_d4ee_4bdd_814c_3de1a07e2dc5.slice/crio-fe903b8ad4f61fb0dc27815c99c68d801b3993bf8dad797769bafa650f37d756 WatchSource:0}: Error finding container fe903b8ad4f61fb0dc27815c99c68d801b3993bf8dad797769bafa650f37d756: Status 404 returned error can't find the container with id fe903b8ad4f61fb0dc27815c99c68d801b3993bf8dad797769bafa650f37d756
Oct 30 19:31:00.353583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.353557    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"]
Oct 30 19:31:00.355907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.355866    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:00.356003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.355957    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:00.356043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.355998    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:00.662689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.662592    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" event={"ID":"e498f782-d4ee-4bdd-814c-3de1a07e2dc5","Type":"ContainerStarted","Data":"106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce"}
Oct 30 19:31:00.662689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.662668    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" event={"ID":"e498f782-d4ee-4bdd-814c-3de1a07e2dc5","Type":"ContainerStarted","Data":"fe903b8ad4f61fb0dc27815c99c68d801b3993bf8dad797769bafa650f37d756"}
Oct 30 19:31:00.663036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.662988    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:31:00.664219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.664194    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:31:00.732492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:00.714487    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podStartSLOduration=1.714463992 podStartE2EDuration="1.714463992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:00.687323138 +0000 UTC m=+1442.684238557" watchObservedRunningTime="2026-10-30 19:31:00.714463992 +0000 UTC m=+1442.711379376"
Oct 30 19:31:01.049321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:01.049284    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:31:01.051106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:01.051051    6777 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:31:01.051273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:01.051132    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:31:02.063083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.053508    6777 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:31:02.063083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.053582    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:31:02.315826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.315742    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"]
Oct 30 19:31:02.319831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.319805    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:02.328431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.326837    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-69f3a9\""
Oct 30 19:31:02.328431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.326878    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-69f3a9-dockercfg-jgpj2\""
Oct 30 19:31:02.331303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.331271    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"]
Oct 30 19:31:02.392694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.392659    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"]
Oct 30 19:31:02.397093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.396934    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:02.398574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.398541    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"]
Oct 30 19:31:02.407516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.407483    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2176303e-2380-47ad-81c5-afba1009961d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn\" (UID: \"2176303e-2380-47ad-81c5-afba1009961d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:02.409306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.409277    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2c8f\" (UniqueName: \"kubernetes.io/projected/2176303e-2380-47ad-81c5-afba1009961d-kube-api-access-p2c8f\") pod \"payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn\" (UID: \"2176303e-2380-47ad-81c5-afba1009961d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:02.538080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.536298    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2176303e-2380-47ad-81c5-afba1009961d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn\" (UID: \"2176303e-2380-47ad-81c5-afba1009961d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:02.538080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.536670    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2c8f\" (UniqueName: \"kubernetes.io/projected/2176303e-2380-47ad-81c5-afba1009961d-kube-api-access-p2c8f\") pod \"payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn\" (UID: \"2176303e-2380-47ad-81c5-afba1009961d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:02.577884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.559920    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2176303e-2380-47ad-81c5-afba1009961d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn\" (UID: \"2176303e-2380-47ad-81c5-afba1009961d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:02.668084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.667521    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2prdk\" (UniqueName: \"kubernetes.io/projected/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-kube-api-access-2prdk\") pod \"payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54\" (UID: \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:02.668084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.667574    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54\" (UID: \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:02.684236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.669154    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2c8f\" (UniqueName: \"kubernetes.io/projected/2176303e-2380-47ad-81c5-afba1009961d-kube-api-access-p2c8f\") pod \"payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn\" (UID: \"2176303e-2380-47ad-81c5-afba1009961d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:02.784107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.773607    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2prdk\" (UniqueName: \"kubernetes.io/projected/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-kube-api-access-2prdk\") pod \"payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54\" (UID: \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:02.784107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.773679    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54\" (UID: \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:02.799919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.799885    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54\" (UID: \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:02.809253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.808768    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2prdk\" (UniqueName: \"kubernetes.io/projected/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-kube-api-access-2prdk\") pod \"payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54\" (UID: \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:02.944640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:02.944552    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:03.016357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.015960    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:03.071409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.055479    6777 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:31:03.071409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.055546    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:31:03.294317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.294262    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"]
Oct 30 19:31:03.381311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.381276    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"]
Oct 30 19:31:03.384734 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:03.384700    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc9bd5f_9cac_4cdb_861a_006ec25036fd.slice/crio-ebc1116369d42381bb38ac0747787287cf53f5e16eb9bbced9e88b522f22429c WatchSource:0}: Error finding container ebc1116369d42381bb38ac0747787287cf53f5e16eb9bbced9e88b522f22429c: Status 404 returned error can't find the container with id ebc1116369d42381bb38ac0747787287cf53f5e16eb9bbced9e88b522f22429c
Oct 30 19:31:03.765371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.765004    6777 patch_prober.go:29] interesting pod/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:31:03.765371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.765077    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:31:03.786964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.786915    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn" event={"ID":"2176303e-2380-47ad-81c5-afba1009961d","Type":"ContainerStarted","Data":"2c4be0ea519487ec959908a4e0f3259c59525ca4dae260e248b53e01de0f5217"}
Oct 30 19:31:03.787281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.787254    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn" event={"ID":"2176303e-2380-47ad-81c5-afba1009961d","Type":"ContainerStarted","Data":"d2dd0cc49fbedb115514889ee1152598ba945e1715916e9621d08f84d8998572"}
Oct 30 19:31:03.788651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.788630    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:03.791214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.791157    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" event={"ID":"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78","Type":"ContainerDied","Data":"5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db"}
Oct 30 19:31:03.791681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.790999    6777 generic.go:356] "Generic (PLEG): container finished" podID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerID="5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db" exitCode=0
Oct 30 19:31:03.795086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.794764    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54" event={"ID":"dfc9bd5f-9cac-4cdb-861a-006ec25036fd","Type":"ContainerStarted","Data":"ebc1116369d42381bb38ac0747787287cf53f5e16eb9bbced9e88b522f22429c"}
Oct 30 19:31:03.795748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.795719    6777 scope.go:122] "RemoveContainer" containerID="5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db"
Oct 30 19:31:03.799987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.799948    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:03.800278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.800250    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:03.800533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.800504    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:03.899174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:03.893426    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn" podStartSLOduration=1.893402308 podStartE2EDuration="1.893402308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:03.817311823 +0000 UTC m=+1445.814227211" watchObservedRunningTime="2026-10-30 19:31:03.893402308 +0000 UTC m=+1445.890317694"
Oct 30 19:31:04.051765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:04.051733    6777 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:31:04.052009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:04.051980    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:31:04.812526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:04.812486    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" event={"ID":"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78","Type":"ContainerStarted","Data":"4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af"}
Oct 30 19:31:04.815924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:04.815894    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54" event={"ID":"dfc9bd5f-9cac-4cdb-861a-006ec25036fd","Type":"ContainerStarted","Data":"847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd"}
Oct 30 19:31:04.816200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:04.816185    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:05.001239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:04.993577    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54" podStartSLOduration=2.993554048 podStartE2EDuration="2.993554048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:04.967983401 +0000 UTC m=+1446.964898786" watchObservedRunningTime="2026-10-30 19:31:04.993554048 +0000 UTC m=+1446.990469435"
Oct 30 19:31:05.016822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:05.016786    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:31:05.017208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:05.017183    6777 patch_prober.go:29] interesting pod/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:31:05.017391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:05.017365    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:31:05.087183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:05.086642    6777 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:31:05.087183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:05.086702    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:31:05.136845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:05.136793    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:31:05.716522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:05.716489    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:31:05.857784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:05.857743    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:05.926080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:05.925247    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:06.024545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:06.024445    6777 patch_prober.go:29] interesting pod/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:31:06.025032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:06.025001    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:31:06.053650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:06.053614    6777 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:31:06.054000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:06.053971    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:31:06.209652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:06.209619    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:31:06.456081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:06.450405    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"]
Oct 30 19:31:06.937131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:06.937099    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:06.944425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:06.944389    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"]
Oct 30 19:31:06.964867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:06.964838    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c86696-openshift-default-dockercfg-zhj6j\""
Oct 30 19:31:07.002811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.001986    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-certs\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.002811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.002084    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7jm\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-kube-api-access-rt7jm\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.002811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.002115    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-envoy\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.002811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.002187    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-data\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.002811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.002230    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-podinfo\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.002811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.002311    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-socket\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.002811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.002467    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-token\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.002811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.002517    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-credential-socket\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.002811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.002559    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcc116f-353d-4ba8-836b-ca158e68bac4-istiod-ca-cert\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.025365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.024547    6777 patch_prober.go:29] interesting pod/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:31:07.025365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.024613    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:31:07.058084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.052476    6777 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:31:07.058084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.052536    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:31:07.105080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.104215    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7jm\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-kube-api-access-rt7jm\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.105080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.104277    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-envoy\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.105080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.104351    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-data\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.105080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.104394    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-podinfo\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.105080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.104478    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-socket\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.105080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.104631    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-token\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.105080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.104690    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-credential-socket\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.105080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.104737    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcc116f-353d-4ba8-836b-ca158e68bac4-istiod-ca-cert\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.105080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.104875    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-certs\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.105711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.105418    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-certs\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.113099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.106389    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-socket\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.113099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.106649    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-data\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.113099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.111427    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-credential-socket\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.113099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.112445    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcc116f-353d-4ba8-836b-ca158e68bac4-istiod-ca-cert\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.118711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.118300    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-envoy\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.122875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.122513    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-podinfo\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.122875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.122824    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7jm\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-kube-api-access-rt7jm\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.130449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.129711    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-token\") pod \"e2e-inf-b-c86696-openshift-default-859496475c-srmqr\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") " pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.404015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.403974    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:07.944975 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.944943    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"]
Oct 30 19:31:07.947382 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:07.947359    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcc116f_353d_4ba8_836b_ca158e68bac4.slice/crio-32254d5eab7b2597aeb3f399988910cb3c57b207c40b3a6f248ea45bd060870c WatchSource:0}: Error finding container 32254d5eab7b2597aeb3f399988910cb3c57b207c40b3a6f248ea45bd060870c: Status 404 returned error can't find the container with id 32254d5eab7b2597aeb3f399988910cb3c57b207c40b3a6f248ea45bd060870c
Oct 30 19:31:07.950569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.950495    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:07.951132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.951105    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:07.951311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:07.951291    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:08.019885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:08.018350    6777 patch_prober.go:29] interesting pod/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:31:08.019885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:08.018417    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:31:08.025736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:08.025706    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" event={"ID":"9dcc116f-353d-4ba8-836b-ca158e68bac4","Type":"ContainerStarted","Data":"32254d5eab7b2597aeb3f399988910cb3c57b207c40b3a6f248ea45bd060870c"}
Oct 30 19:31:08.078105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:08.060539    6777 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 19:31:08.078105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:08.060605    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 19:31:09.012753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:09.012662    6777 patch_prober.go:29] interesting pod/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:31:09.012753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:09.012732    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:31:09.036453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:09.036419    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" event={"ID":"9dcc116f-353d-4ba8-836b-ca158e68bac4","Type":"ContainerStarted","Data":"71ef753f4bae2c2cc33b7646002cf92a65e1fc90d3666a0c8a34fc20876231dd"}
Oct 30 19:31:09.080484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:09.080423    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" podStartSLOduration=3.080404753 podStartE2EDuration="3.080404753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:09.066521507 +0000 UTC m=+1451.063436891" watchObservedRunningTime="2026-10-30 19:31:09.080404753 +0000 UTC m=+1451.077320140"
Oct 30 19:31:09.406243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:09.406212    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:09.414420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:09.414389    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:09.609183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:09.609149    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:31:09.609613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:09.609590    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:31:09.625923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:09.625892    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:31:10.012793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.012755    6777 patch_prober.go:29] interesting pod/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:31:10.013239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.012813    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:31:10.044752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.044717    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:10.045737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.045717    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:31:10.539428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.539397    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"]
Oct 30 19:31:10.544892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.544868    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.580510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.580483    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ef5dca-openshift-default-dockercfg-vkm6l\""
Oct 30 19:31:10.591559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.588993    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"]
Oct 30 19:31:10.623984 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.623949    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-envoy\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.623984 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.623986    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-credential-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.624342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.624150    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-certs\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.624342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.624214    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-data\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.624457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.624379    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8120b5d3-a576-47cc-a4ff-6b698af54051-istiod-ca-cert\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.624520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.624485    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkqw9\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-kube-api-access-qkqw9\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.624576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.624550    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-podinfo\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.624694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.624678    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-token\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.624757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.624712    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726127    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8120b5d3-a576-47cc-a4ff-6b698af54051-istiod-ca-cert\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726244    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkqw9\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-kube-api-access-qkqw9\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726267    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-podinfo\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726308    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-token\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726350    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726484    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-envoy\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726509    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-credential-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726576    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-certs\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726600    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-data\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726858    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.726761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.726916    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-credential-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.727203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.727015    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8120b5d3-a576-47cc-a4ff-6b698af54051-istiod-ca-cert\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.727203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.727048    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-certs\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.727203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.727126    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-data\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.729133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.729111    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-podinfo\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.729291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.729271    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-envoy\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.738769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.738741    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-token\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.738867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.738816    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkqw9\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-kube-api-access-qkqw9\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:10.901345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:10.901249    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:11.031206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:11.029980    6777 patch_prober.go:29] interesting pod/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 19:31:11.031206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:11.030127    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 19:31:11.225845 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:11.225815    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8120b5d3_a576_47cc_a4ff_6b698af54051.slice/crio-ac40c09e98fc1b445f7ddd8ea83a2c2232f1736459008eac9c0fbdead4c2574e WatchSource:0}: Error finding container ac40c09e98fc1b445f7ddd8ea83a2c2232f1736459008eac9c0fbdead4c2574e: Status 404 returned error can't find the container with id ac40c09e98fc1b445f7ddd8ea83a2c2232f1736459008eac9c0fbdead4c2574e
Oct 30 19:31:11.225984 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:11.225866    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"]
Oct 30 19:31:11.229650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:11.229601    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:11.230430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:11.229678    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:11.230430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:11.229726    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:12.018751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.018722    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:31:12.019307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.019283    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:31:12.020427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.020405    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:31:12.103180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.103095    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" event={"ID":"8120b5d3-a576-47cc-a4ff-6b698af54051","Type":"ContainerStarted","Data":"26c846b3c6568675dced39f56dd25fbbd24cd4ed0444ac66515be5b86cfddf7a"}
Oct 30 19:31:12.108102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.103644    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" event={"ID":"8120b5d3-a576-47cc-a4ff-6b698af54051","Type":"ContainerStarted","Data":"ac40c09e98fc1b445f7ddd8ea83a2c2232f1736459008eac9c0fbdead4c2574e"}
Oct 30 19:31:12.138392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.138325    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" podStartSLOduration=2.138306498 podStartE2EDuration="2.138306498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:12.135521327 +0000 UTC m=+1454.132436713" watchObservedRunningTime="2026-10-30 19:31:12.138306498 +0000 UTC m=+1454.135221882"
Oct 30 19:31:12.582797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.582764    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"]
Oct 30 19:31:12.587036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.587007    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"]
Oct 30 19:31:12.587234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.587094    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:12.593270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.591378    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:12.594169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.594008    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c86696\""
Oct 30 19:31:12.598535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.598509    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"]
Oct 30 19:31:12.620962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.620927    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"]
Oct 30 19:31:12.694637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.694595    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66nbj\" (UniqueName: \"kubernetes.io/projected/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-kube-api-access-66nbj\") pod \"payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8\" (UID: \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:12.694834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.694697    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfzp\" (UniqueName: \"kubernetes.io/projected/eaf2532e-7980-44d9-b14e-650e57d53c1c-kube-api-access-4cfzp\") pod \"payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8\" (UID: \"eaf2532e-7980-44d9-b14e-650e57d53c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:12.694902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.694831    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8\" (UID: \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:12.695030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.695010    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2532e-7980-44d9-b14e-650e57d53c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8\" (UID: \"eaf2532e-7980-44d9-b14e-650e57d53c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:12.798739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.798345    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66nbj\" (UniqueName: \"kubernetes.io/projected/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-kube-api-access-66nbj\") pod \"payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8\" (UID: \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:12.798739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.798401    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfzp\" (UniqueName: \"kubernetes.io/projected/eaf2532e-7980-44d9-b14e-650e57d53c1c-kube-api-access-4cfzp\") pod \"payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8\" (UID: \"eaf2532e-7980-44d9-b14e-650e57d53c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:12.798739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.798482    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8\" (UID: \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:12.798739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.798607    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2532e-7980-44d9-b14e-650e57d53c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8\" (UID: \"eaf2532e-7980-44d9-b14e-650e57d53c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:12.814965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.799564    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2532e-7980-44d9-b14e-650e57d53c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8\" (UID: \"eaf2532e-7980-44d9-b14e-650e57d53c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:12.814965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.800804    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8\" (UID: \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:12.865129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.863073    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfzp\" (UniqueName: \"kubernetes.io/projected/eaf2532e-7980-44d9-b14e-650e57d53c1c-kube-api-access-4cfzp\") pod \"payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8\" (UID: \"eaf2532e-7980-44d9-b14e-650e57d53c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:12.940270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.926260    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:12.940270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.926498    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:12.948551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.948326    6777 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:31:12.948551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:12.948389    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 19:31:13.060173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:13.047324    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66nbj\" (UniqueName: \"kubernetes.io/projected/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-kube-api-access-66nbj\") pod \"payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8\" (UID: \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:13.233117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:13.231868    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:13.496966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:13.496939    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"]
Oct 30 19:31:13.633680 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:13.632653    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e1fddb_9a89_4d9d_a730_7d063e6fa0d0.slice/crio-e13d08caf64283b00e8d62bba5b58a5ec55813a2884e24315a563ede6f503798 WatchSource:0}: Error finding container e13d08caf64283b00e8d62bba5b58a5ec55813a2884e24315a563ede6f503798: Status 404 returned error can't find the container with id e13d08caf64283b00e8d62bba5b58a5ec55813a2884e24315a563ede6f503798
Oct 30 19:31:13.634191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:13.634166    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"]
Oct 30 19:31:13.680294 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:31:13.680258    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:31:13.917082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:13.909396    6777 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:31:13.917082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:13.909454    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 19:31:14.192902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.192787    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8" event={"ID":"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0","Type":"ContainerStarted","Data":"cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954"}
Oct 30 19:31:14.192902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.192872    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8" event={"ID":"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0","Type":"ContainerStarted","Data":"e13d08caf64283b00e8d62bba5b58a5ec55813a2884e24315a563ede6f503798"}
Oct 30 19:31:14.194005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.193982    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:14.196479 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.196449    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8" event={"ID":"eaf2532e-7980-44d9-b14e-650e57d53c1c","Type":"ContainerStarted","Data":"b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052"}
Oct 30 19:31:14.196587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.196498    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8" event={"ID":"eaf2532e-7980-44d9-b14e-650e57d53c1c","Type":"ContainerStarted","Data":"acc7586121de3bad6dcb8ec91b7fbb1488d6d03299d716c1b96c1a93fa9221b7"}
Oct 30 19:31:14.197232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.197213    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:14.218047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.218000    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8" podStartSLOduration=2.217986613 podStartE2EDuration="2.217986613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:14.213240897 +0000 UTC m=+1456.210156283" watchObservedRunningTime="2026-10-30 19:31:14.217986613 +0000 UTC m=+1456.214902032"
Oct 30 19:31:14.232514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.232458    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8" podStartSLOduration=2.232441594 podStartE2EDuration="2.232441594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:14.231566088 +0000 UTC m=+1456.228481473" watchObservedRunningTime="2026-10-30 19:31:14.232441594 +0000 UTC m=+1456.229356988"
Oct 30 19:31:14.909082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.907834    6777 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:31:14.909082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:14.907900    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 19:31:15.895563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:15.895535    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:31:15.901805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:15.901779    6777 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:31:15.901941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:15.901828    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 19:31:16.902355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:16.902316    6777 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:31:16.902863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:16.902388    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 19:31:17.020318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:17.019569    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:31:17.906575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:17.906540    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:17.907047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:17.907023    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:17.908020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:17.908001    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:31:21.752199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:21.752162    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"]
Oct 30 19:31:21.752750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:21.752422    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg" podUID="384ea436-9ae8-493f-94ee-2af351c41196" containerName="maas-api" containerID="cri-o://a2be1eca4691db6ac9dd7a5700bb58c20f20a7b9bec732e7a37bfbd6a42ebf9b" gracePeriod=30
Oct 30 19:31:22.284025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.283992    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"]
Oct 30 19:31:22.284290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.284272    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e62-x8jbc"]
Oct 30 19:31:22.306147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.299351    6777 generic.go:356] "Generic (PLEG): container finished" podID="384ea436-9ae8-493f-94ee-2af351c41196" containerID="a2be1eca4691db6ac9dd7a5700bb58c20f20a7b9bec732e7a37bfbd6a42ebf9b" exitCode=0
Oct 30 19:31:22.306147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.299477    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg" event={"ID":"384ea436-9ae8-493f-94ee-2af351c41196","Type":"ContainerDied","Data":"a2be1eca4691db6ac9dd7a5700bb58c20f20a7b9bec732e7a37bfbd6a42ebf9b"}
Oct 30 19:31:22.372221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.372188    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:31:22.437022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.436981    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384ea436-9ae8-493f-94ee-2af351c41196-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384ea436-9ae8-493f-94ee-2af351c41196-openshift-service-ca\") pod \"384ea436-9ae8-493f-94ee-2af351c41196\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") "
Oct 30 19:31:22.437242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.437158    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-metrics-certs\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-metrics-certs\") pod \"384ea436-9ae8-493f-94ee-2af351c41196\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") "
Oct 30 19:31:22.437385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.437363    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384ea436-9ae8-493f-94ee-2af351c41196-kube-api-access-8n9rs\" (UniqueName: \"kubernetes.io/projected/384ea436-9ae8-493f-94ee-2af351c41196-kube-api-access-8n9rs\") pod \"384ea436-9ae8-493f-94ee-2af351c41196\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") "
Oct 30 19:31:22.437450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.437410    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384ea436-9ae8-493f-94ee-2af351c41196-openshift-service-ca" pod "384ea436-9ae8-493f-94ee-2af351c41196" (UID: "384ea436-9ae8-493f-94ee-2af351c41196"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:22.437540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.437525    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls\") pod \"384ea436-9ae8-493f-94ee-2af351c41196\" (UID: \"384ea436-9ae8-493f-94ee-2af351c41196\") "
Oct 30 19:31:22.438262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.438239    6777 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384ea436-9ae8-493f-94ee-2af351c41196-openshift-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:22.439600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.439573    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384ea436-9ae8-493f-94ee-2af351c41196-kube-api-access-8n9rs" pod "384ea436-9ae8-493f-94ee-2af351c41196" (UID: "384ea436-9ae8-493f-94ee-2af351c41196"). InnerVolumeSpecName "kube-api-access-8n9rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:22.439711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.439616    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-metrics-certs" pod "384ea436-9ae8-493f-94ee-2af351c41196" (UID: "384ea436-9ae8-493f-94ee-2af351c41196"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:22.439864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.439843    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls" pod "384ea436-9ae8-493f-94ee-2af351c41196" (UID: "384ea436-9ae8-493f-94ee-2af351c41196"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:22.539754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.539671    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-maas-api-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:22.539754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.539697    6777 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/384ea436-9ae8-493f-94ee-2af351c41196-metrics-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:22.539754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:22.539707    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9rs\" (UniqueName: \"kubernetes.io/projected/384ea436-9ae8-493f-94ee-2af351c41196-kube-api-access-8n9rs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:23.311815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:23.311776    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"
Oct 30 19:31:23.312354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:23.311810    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg" event={"ID":"384ea436-9ae8-493f-94ee-2af351c41196","Type":"ContainerDied","Data":"b785d983c98d393912a4eb5507790c3ceb0a3b9a4729ec8a7953b363e8dbb795"}
Oct 30 19:31:23.312354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:23.311882    6777 scope.go:122] "RemoveContainer" containerID="a2be1eca4691db6ac9dd7a5700bb58c20f20a7b9bec732e7a37bfbd6a42ebf9b"
Oct 30 19:31:23.340964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:23.340929    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"]
Oct 30 19:31:23.349211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:23.349183    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f61e62-66b66896cc-hxfqg"]
Oct 30 19:31:24.116467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:24.116435    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384ea436-9ae8-493f-94ee-2af351c41196" path="/var/lib/kubelet/pods/384ea436-9ae8-493f-94ee-2af351c41196/volumes"
Oct 30 19:31:24.117001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:24.116987    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7232cee-b66b-49fb-b789-e7a814bfc2a7" path="/var/lib/kubelet/pods/a7232cee-b66b-49fb-b789-e7a814bfc2a7/volumes"
Oct 30 19:31:26.220990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:26.220957    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:31:26.221476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:26.221134    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:31:27.351370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:27.351324    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"]
Oct 30 19:31:27.353550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:27.353508    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4012a-c5mw7"]
Oct 30 19:31:28.115400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:28.115366    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8c65cf-f75b-40d5-85e4-f07d93b82995" path="/var/lib/kubelet/pods/0d8c65cf-f75b-40d5-85e4-f07d93b82995/volumes"
Oct 30 19:31:29.549897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:29.549857    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:31:30.298957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.298917    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"]
Oct 30 19:31:30.299789 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.299773    6777 cpu_manager.go:414] "containerMap: removing container" podUID="384ea436-9ae8-493f-94ee-2af351c41196" containerName="maas-api"
Oct 30 19:31:30.299851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.299791    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384ea436-9ae8-493f-94ee-2af351c41196" containerName="maas-api"
Oct 30 19:31:30.299913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.299903    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="384ea436-9ae8-493f-94ee-2af351c41196" containerName="maas-api"
Oct 30 19:31:30.303644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.303617    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.317196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.317165    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-afdf16-openshift-default-dockercfg-x4cz2\""
Oct 30 19:31:30.322468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.322440    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"]
Oct 30 19:31:30.492863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.492825    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-data\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.493068 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.493009    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxlr\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-kube-api-access-vcxlr\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.493068 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.493043    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istiod-ca-cert\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.493170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.493124    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-envoy\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.493262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.493248    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-certs\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.493331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.493322    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-podinfo\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.493424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.493397    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.493492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.493439    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-token\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.493575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.493552    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-credential-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.603141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.602519    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxlr\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-kube-api-access-vcxlr\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.603141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.602586    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istiod-ca-cert\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.603141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.602652    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-envoy\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.603141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.602757    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-certs\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.603141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.602859    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-podinfo\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.603141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.602896    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.603141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.602923    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-token\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.603141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.603021    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-credential-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.604001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.603161    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-data\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.604001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.603753    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-data\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.613082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.604878    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istiod-ca-cert\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.613082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.606398    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.613082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.606656    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-certs\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.613082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.607364    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-credential-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.613082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.610957    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-podinfo\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.674079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.665895    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxlr\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-kube-api-access-vcxlr\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.678079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.674448    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-envoy\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.678079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.676917    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-token\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:30.928762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:30.928670    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:31.984081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:31.982384    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"]
Oct 30 19:31:31.986808 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:31.986768    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c6e626_bfc1_47cf_bb9a_5828cb65ac8c.slice/crio-e400f237387537b9921eeeac6d1fc2b8bee844a8841f29c9b32c50601ff6c98c WatchSource:0}: Error finding container e400f237387537b9921eeeac6d1fc2b8bee844a8841f29c9b32c50601ff6c98c: Status 404 returned error can't find the container with id e400f237387537b9921eeeac6d1fc2b8bee844a8841f29c9b32c50601ff6c98c
Oct 30 19:31:31.989849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:31.989817    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:31.989919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:31.989891    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:31.989964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:31.989926    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:32.281181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:32.281142    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"]
Oct 30 19:31:32.282083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:32.282027    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy" containerID="cri-o://63c138333167032010fd1a986dfc68804eeb5e583b544fff8395abfcd3a62d30" gracePeriod=30
Oct 30 19:31:32.510437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:32.510393    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" event={"ID":"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c","Type":"ContainerStarted","Data":"89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709"}
Oct 30 19:31:32.510738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:32.510718    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" event={"ID":"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c","Type":"ContainerStarted","Data":"e400f237387537b9921eeeac6d1fc2b8bee844a8841f29c9b32c50601ff6c98c"}
Oct 30 19:31:32.930222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:32.930181    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:33.597496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.597438    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" podStartSLOduration=3.597420454 podStartE2EDuration="3.597420454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:32.562477545 +0000 UTC m=+1474.559392932" watchObservedRunningTime="2026-10-30 19:31:33.597420454 +0000 UTC m=+1475.594335836"
Oct 30 19:31:33.599615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.599590    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"]
Oct 30 19:31:33.605465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.605437    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.621939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.621895    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c24c01-openshift-default-dockercfg-6xcnv\""
Oct 30 19:31:33.628783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.628736    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"]
Oct 30 19:31:33.721091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.720840    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjmp\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-kube-api-access-4xjmp\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.721091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.720989    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-envoy\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.721388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.721141    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.721388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.721239    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-credential-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.721388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.721342    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-data\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.721388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.721377    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-token\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.721684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.721425    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-podinfo\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.721684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.721636    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc320727-adce-4d5f-bab8-3b6806c955e0-istiod-ca-cert\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.722492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.721700    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-certs\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.828652    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-podinfo\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.828812    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc320727-adce-4d5f-bab8-3b6806c955e0-istiod-ca-cert\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.828863    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-certs\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.828903    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjmp\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-kube-api-access-4xjmp\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.829001    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-envoy\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.829139    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.829188    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-credential-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.829273    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-data\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.829306    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-token\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.830667    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-credential-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.830882    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.836646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.835421    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-certs\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.849223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.843254    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc320727-adce-4d5f-bab8-3b6806c955e0-istiod-ca-cert\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.860142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.858531    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-data\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.878080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.870827    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-token\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.878080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.871126    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-envoy\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.878080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.871650    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-podinfo\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.881622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.881554    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjmp\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-kube-api-access-4xjmp\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.942439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.942399    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:33.948843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.948543    6777 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:31:33.949153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:33.949121    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 19:31:34.448156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:34.448120    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"]
Oct 30 19:31:34.477104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:34.477049    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:34.477292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:34.477157    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:34.477292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:34.477204    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:34.542667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:34.542607    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" event={"ID":"cc320727-adce-4d5f-bab8-3b6806c955e0","Type":"ContainerStarted","Data":"d3411d8a97ea31b04360a7b275f52a866bcac16797698dc90d10cc7b5a60723d"}
Oct 30 19:31:34.929874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:34.929835    6777 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:31:34.930342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:34.929905    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 19:31:35.587077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:35.587026    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" event={"ID":"cc320727-adce-4d5f-bab8-3b6806c955e0","Type":"ContainerStarted","Data":"1757d769dc961492da3aaeaa70127548e1e1e506bc4ddd83dd248895377fcdf0"}
Oct 30 19:31:35.930041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:35.929946    6777 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:31:35.930041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:35.930010    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 19:31:35.946791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:35.946757    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:35.978360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:35.978318    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:36.010177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:36.010120    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" podStartSLOduration=3.010104284 podStartE2EDuration="3.010104284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:35.615875738 +0000 UTC m=+1477.612791122" watchObservedRunningTime="2026-10-30 19:31:36.010104284 +0000 UTC m=+1478.007019669"
Oct 30 19:31:36.373321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:36.373275    6777 patch_prober.go:29] interesting pod/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:31:36.373616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:36.373353    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:31:36.597238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:36.597196    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:36.598153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:36.598128    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:31:36.930136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:36.930097    6777 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:31:36.930575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:36.930154    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 19:31:37.615624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.615538    6777 generic.go:356] "Generic (PLEG): container finished" podID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerID="63c138333167032010fd1a986dfc68804eeb5e583b544fff8395abfcd3a62d30" exitCode=0
Oct 30 19:31:37.615624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.615611    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" event={"ID":"4e939f83-c3c3-424c-a26b-a20a197950ec","Type":"ContainerDied","Data":"63c138333167032010fd1a986dfc68804eeb5e583b544fff8395abfcd3a62d30"}
Oct 30 19:31:37.615847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.615676    6777 scope.go:122] "RemoveContainer" containerID="cb4234e3e81b3ea5aa9bc06cc0e4878c43fca4f73c5589deb56bc90ed341d898"
Oct 30 19:31:37.697320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.697008    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:31:37.711037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.710990    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"]
Oct 30 19:31:37.711368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.711294    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" containerID="cri-o://4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af" gracePeriod=30
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.744944    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-credential-socket\") pod \"4e939f83-c3c3-424c-a26b-a20a197950ec\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") "
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.745039    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-token\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-token\") pod \"4e939f83-c3c3-424c-a26b-a20a197950ec\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") "
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.745361    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-credential-socket" pod "4e939f83-c3c3-424c-a26b-a20a197950ec" (UID: "4e939f83-c3c3-424c-a26b-a20a197950ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.745591    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-podinfo\") pod \"4e939f83-c3c3-424c-a26b-a20a197950ec\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") "
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.745653    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-data\") pod \"4e939f83-c3c3-424c-a26b-a20a197950ec\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") "
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.745982    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-data" pod "4e939f83-c3c3-424c-a26b-a20a197950ec" (UID: "4e939f83-c3c3-424c-a26b-a20a197950ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.746350    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-envoy\") pod \"4e939f83-c3c3-424c-a26b-a20a197950ec\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") "
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.746430    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-certs\") pod \"4e939f83-c3c3-424c-a26b-a20a197950ec\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") "
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.746705    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-certs" pod "4e939f83-c3c3-424c-a26b-a20a197950ec" (UID: "4e939f83-c3c3-424c-a26b-a20a197950ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.746921    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e939f83-c3c3-424c-a26b-a20a197950ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e939f83-c3c3-424c-a26b-a20a197950ec-istiod-ca-cert\") pod \"4e939f83-c3c3-424c-a26b-a20a197950ec\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") "
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.747289    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e939f83-c3c3-424c-a26b-a20a197950ec-istiod-ca-cert" pod "4e939f83-c3c3-424c-a26b-a20a197950ec" (UID: "4e939f83-c3c3-424c-a26b-a20a197950ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.747379    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-kube-api-access-phkvb\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-kube-api-access-phkvb\") pod \"4e939f83-c3c3-424c-a26b-a20a197950ec\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") "
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.747414    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-socket\") pod \"4e939f83-c3c3-424c-a26b-a20a197950ec\" (UID: \"4e939f83-c3c3-424c-a26b-a20a197950ec\") "
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.747859    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-socket" pod "4e939f83-c3c3-424c-a26b-a20a197950ec" (UID: "4e939f83-c3c3-424c-a26b-a20a197950ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.748839    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.748863    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.748877    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.748891    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:37.766091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.748906    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e939f83-c3c3-424c-a26b-a20a197950ec-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:37.785277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.784727    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-token" pod "4e939f83-c3c3-424c-a26b-a20a197950ec" (UID: "4e939f83-c3c3-424c-a26b-a20a197950ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:37.787524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.787384    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-kube-api-access-phkvb" pod "4e939f83-c3c3-424c-a26b-a20a197950ec" (UID: "4e939f83-c3c3-424c-a26b-a20a197950ec"). InnerVolumeSpecName "kube-api-access-phkvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:37.788428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.788381    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-podinfo" pod "4e939f83-c3c3-424c-a26b-a20a197950ec" (UID: "4e939f83-c3c3-424c-a26b-a20a197950ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:31:37.788937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.788908    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-envoy" pod "4e939f83-c3c3-424c-a26b-a20a197950ec" (UID: "4e939f83-c3c3-424c-a26b-a20a197950ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:37.879142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.856743    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phkvb\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-kube-api-access-phkvb\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:37.879142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.856779    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:37.879142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.856796    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:37.879142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.856811    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e939f83-c3c3-424c-a26b-a20a197950ec-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:37.995089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.995006    6777 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:31:37.995089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:37.995076    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 19:31:38.228173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.228113    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"]
Oct 30 19:31:38.230178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.230147    6777 cpu_manager.go:414] "containerMap: removing container" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy"
Oct 30 19:31:38.230178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.230176    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy"
Oct 30 19:31:38.230381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.230216    6777 cpu_manager.go:414] "containerMap: removing container" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy"
Oct 30 19:31:38.230381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.230225    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy"
Oct 30 19:31:38.230707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.230689    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy"
Oct 30 19:31:38.230780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.230720    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" containerName="istio-proxy"
Oct 30 19:31:38.246092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.246050    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.246246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.246093    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"]
Oct 30 19:31:38.271535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.271506    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:31:38.271935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.271890    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:31:38.310034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.309998    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.310332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.310309    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.310694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.310672    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb7jj\" (UniqueName: \"kubernetes.io/projected/61f7e246-4210-4dd8-bd1a-775204614735-kube-api-access-fb7jj\") pod \"maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.412450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.412401    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb7jj\" (UniqueName: \"kubernetes.io/projected/61f7e246-4210-4dd8-bd1a-775204614735-kube-api-access-fb7jj\") pod \"maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.412648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.412507    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.412648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.412633    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.413370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.413345    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.415099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.415077    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.426692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.426650    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb7jj\" (UniqueName: \"kubernetes.io/projected/61f7e246-4210-4dd8-bd1a-775204614735-kube-api-access-fb7jj\") pod \"maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.592673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.592619    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:38.626798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.626756    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt" event={"ID":"4e939f83-c3c3-424c-a26b-a20a197950ec","Type":"ContainerDied","Data":"5eade939c24aaae8bc422c36b0550998072ec67eeff0501481309e23a3fd3f24"}
Oct 30 19:31:38.626999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.626841    6777 scope.go:122] "RemoveContainer" containerID="63c138333167032010fd1a986dfc68804eeb5e583b544fff8395abfcd3a62d30"
Oct 30 19:31:38.626999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.626774    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"
Oct 30 19:31:38.747050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.747014    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"]
Oct 30 19:31:38.751211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.751181    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f61e62-openshift-default-6fb69f9447-p8jgt"]
Oct 30 19:31:38.893582 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:38.893549    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f7e246_4210_4dd8_bd1a_775204614735.slice/crio-8e6c865156ff8ca183050bd7de94a1b0a0d7cf824626b3da5f398ab28a66cacb WatchSource:0}: Error finding container 8e6c865156ff8ca183050bd7de94a1b0a0d7cf824626b3da5f398ab28a66cacb: Status 404 returned error can't find the container with id 8e6c865156ff8ca183050bd7de94a1b0a0d7cf824626b3da5f398ab28a66cacb
Oct 30 19:31:38.895681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.895648    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"]
Oct 30 19:31:38.931428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.931394    6777 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:31:38.931595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:38.931456    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 19:31:39.510198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.510108    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"]
Oct 30 19:31:39.515231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.515201    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:39.525107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.525074    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c24c01\""
Oct 30 19:31:39.527934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.527908    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"]
Oct 30 19:31:39.538901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.538864    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"]
Oct 30 19:31:39.543643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.543622    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:39.551248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.551218    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c24c01-dockercfg-k44fk\""
Oct 30 19:31:39.557662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.557622    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ff59f2-0974-4f2a-8bd6-b27052afca3d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr\" (UID: \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:39.557833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.557783    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqn4f\" (UniqueName: \"kubernetes.io/projected/94ff59f2-0974-4f2a-8bd6-b27052afca3d-kube-api-access-wqn4f\") pod \"payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr\" (UID: \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:39.561333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.561301    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"]
Oct 30 19:31:39.720080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.705426    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqn4f\" (UniqueName: \"kubernetes.io/projected/94ff59f2-0974-4f2a-8bd6-b27052afca3d-kube-api-access-wqn4f\") pod \"payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr\" (UID: \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:39.720080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.705493    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s59nh\" (UniqueName: \"kubernetes.io/projected/d4c2cc99-d020-4163-bdbb-0ac595964973-kube-api-access-s59nh\") pod \"payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v\" (UID: \"d4c2cc99-d020-4163-bdbb-0ac595964973\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:39.720080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.705641    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c2cc99-d020-4163-bdbb-0ac595964973-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v\" (UID: \"d4c2cc99-d020-4163-bdbb-0ac595964973\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:39.720080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.705906    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ff59f2-0974-4f2a-8bd6-b27052afca3d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr\" (UID: \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:39.720080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.706981    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ff59f2-0974-4f2a-8bd6-b27052afca3d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr\" (UID: \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:39.782785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.781996    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf" event={"ID":"61f7e246-4210-4dd8-bd1a-775204614735","Type":"ContainerStarted","Data":"89130c7827dfef7baa82e00ebad80c4a7ca7efe4dfc0771530e56b03a7b9ad77"}
Oct 30 19:31:39.782785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.782089    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf" event={"ID":"61f7e246-4210-4dd8-bd1a-775204614735","Type":"ContainerStarted","Data":"8e6c865156ff8ca183050bd7de94a1b0a0d7cf824626b3da5f398ab28a66cacb"}
Oct 30 19:31:39.820085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.816624    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s59nh\" (UniqueName: \"kubernetes.io/projected/d4c2cc99-d020-4163-bdbb-0ac595964973-kube-api-access-s59nh\") pod \"payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v\" (UID: \"d4c2cc99-d020-4163-bdbb-0ac595964973\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:39.820085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.816815    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c2cc99-d020-4163-bdbb-0ac595964973-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v\" (UID: \"d4c2cc99-d020-4163-bdbb-0ac595964973\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:39.820085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.818765    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c2cc99-d020-4163-bdbb-0ac595964973-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v\" (UID: \"d4c2cc99-d020-4163-bdbb-0ac595964973\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:39.820085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.819822    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqn4f\" (UniqueName: \"kubernetes.io/projected/94ff59f2-0974-4f2a-8bd6-b27052afca3d-kube-api-access-wqn4f\") pod \"payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr\" (UID: \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:39.848090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.846725    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:39.879136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.878651    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf" podStartSLOduration=1.878629948 podStartE2EDuration="1.878629948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:39.847352202 +0000 UTC m=+1481.844267587" watchObservedRunningTime="2026-10-30 19:31:39.878629948 +0000 UTC m=+1481.875545335"
Oct 30 19:31:39.963685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.963649    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s59nh\" (UniqueName: \"kubernetes.io/projected/d4c2cc99-d020-4163-bdbb-0ac595964973-kube-api-access-s59nh\") pod \"payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v\" (UID: \"d4c2cc99-d020-4163-bdbb-0ac595964973\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:39.976246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.972603    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:39.976246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.974917    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:39.986112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:39.985632    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:31:40.122359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.122318    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e939f83-c3c3-424c-a26b-a20a197950ec" path="/var/lib/kubelet/pods/4e939f83-c3c3-424c-a26b-a20a197950ec/volumes"
Oct 30 19:31:40.172124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.172090    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:40.274847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.274809    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"]
Oct 30 19:31:40.281401 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:40.281373    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ff59f2_0974_4f2a_8bd6_b27052afca3d.slice/crio-c3a5595df8b4c8a10a94755c8455a981cfa2498cb6108d53d4862da997060c23 WatchSource:0}: Error finding container c3a5595df8b4c8a10a94755c8455a981cfa2498cb6108d53d4862da997060c23: Status 404 returned error can't find the container with id c3a5595df8b4c8a10a94755c8455a981cfa2498cb6108d53d4862da997060c23
Oct 30 19:31:40.460295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.460263    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"]
Oct 30 19:31:40.460576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:40.460554    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c2cc99_d020_4163_bdbb_0ac595964973.slice/crio-4d130c862e505c939fc42fa3af35e1227b2f81cda0c02a2665932d66d6887962 WatchSource:0}: Error finding container 4d130c862e505c939fc42fa3af35e1227b2f81cda0c02a2665932d66d6887962: Status 404 returned error can't find the container with id 4d130c862e505c939fc42fa3af35e1227b2f81cda0c02a2665932d66d6887962
Oct 30 19:31:40.792833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.792788    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v" event={"ID":"d4c2cc99-d020-4163-bdbb-0ac595964973","Type":"ContainerStarted","Data":"0daad8f8cce151da002b86b50c73d8531b4351e0a37a5eeefa80d32497a14481"}
Oct 30 19:31:40.793304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.792898    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:40.793304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.792929    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v" event={"ID":"d4c2cc99-d020-4163-bdbb-0ac595964973","Type":"ContainerStarted","Data":"4d130c862e505c939fc42fa3af35e1227b2f81cda0c02a2665932d66d6887962"}
Oct 30 19:31:40.794627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.794594    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr" event={"ID":"94ff59f2-0974-4f2a-8bd6-b27052afca3d","Type":"ContainerStarted","Data":"80ccaa320fdd167037b02b670177893cc91055fca19b9e375ff2601a45d15e2e"}
Oct 30 19:31:40.794779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.794649    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr" event={"ID":"94ff59f2-0974-4f2a-8bd6-b27052afca3d","Type":"ContainerStarted","Data":"c3a5595df8b4c8a10a94755c8455a981cfa2498cb6108d53d4862da997060c23"}
Oct 30 19:31:40.794779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.794702    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:40.796489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.796460    6777 generic.go:356] "Generic (PLEG): container finished" podID="61f7e246-4210-4dd8-bd1a-775204614735" containerID="89130c7827dfef7baa82e00ebad80c4a7ca7efe4dfc0771530e56b03a7b9ad77" exitCode=0
Oct 30 19:31:40.796610 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.796543    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf" event={"ID":"61f7e246-4210-4dd8-bd1a-775204614735","Type":"ContainerDied","Data":"89130c7827dfef7baa82e00ebad80c4a7ca7efe4dfc0771530e56b03a7b9ad77"}
Oct 30 19:31:40.814532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.814425    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v" podStartSLOduration=1.814407569 podStartE2EDuration="1.814407569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:40.814008288 +0000 UTC m=+1482.810923687" watchObservedRunningTime="2026-10-30 19:31:40.814407569 +0000 UTC m=+1482.811322958"
Oct 30 19:31:40.839571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:40.839519    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr" podStartSLOduration=1.839505849 podStartE2EDuration="1.839505849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:40.835179627 +0000 UTC m=+1482.832095013" watchObservedRunningTime="2026-10-30 19:31:40.839505849 +0000 UTC m=+1482.836421279"
Oct 30 19:31:42.022332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.022293    6777 patch_prober.go:29] interesting pod/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:31:42.022758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.022361    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:31:42.057521 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.057443    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:42.183076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.183022    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-openshift-ca\") pod \"61f7e246-4210-4dd8-bd1a-775204614735\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") "
Oct 30 19:31:42.183298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.183182    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-xks-ca\") pod \"61f7e246-4210-4dd8-bd1a-775204614735\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") "
Oct 30 19:31:42.183425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.183399    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f7e246-4210-4dd8-bd1a-775204614735-kube-api-access-fb7jj\" (UniqueName: \"kubernetes.io/projected/61f7e246-4210-4dd8-bd1a-775204614735-kube-api-access-fb7jj\") pod \"61f7e246-4210-4dd8-bd1a-775204614735\" (UID: \"61f7e246-4210-4dd8-bd1a-775204614735\") "
Oct 30 19:31:42.183908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.183877    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-openshift-ca" pod "61f7e246-4210-4dd8-bd1a-775204614735" (UID: "61f7e246-4210-4dd8-bd1a-775204614735"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:42.184278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.184256    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:42.185631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.185605    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-xks-ca" pod "61f7e246-4210-4dd8-bd1a-775204614735" (UID: "61f7e246-4210-4dd8-bd1a-775204614735"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:42.185788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.185766    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f7e246-4210-4dd8-bd1a-775204614735-kube-api-access-fb7jj" pod "61f7e246-4210-4dd8-bd1a-775204614735" (UID: "61f7e246-4210-4dd8-bd1a-775204614735"). InnerVolumeSpecName "kube-api-access-fb7jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:42.286547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.286517    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f7e246-4210-4dd8-bd1a-775204614735-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:42.286547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.286546    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb7jj\" (UniqueName: \"kubernetes.io/projected/61f7e246-4210-4dd8-bd1a-775204614735-kube-api-access-fb7jj\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:42.815740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.815707    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf" event={"ID":"61f7e246-4210-4dd8-bd1a-775204614735","Type":"ContainerDied","Data":"8e6c865156ff8ca183050bd7de94a1b0a0d7cf824626b3da5f398ab28a66cacb"}
Oct 30 19:31:42.815740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.815731    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"
Oct 30 19:31:42.815960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:42.815765    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e6c865156ff8ca183050bd7de94a1b0a0d7cf824626b3da5f398ab28a66cacb"
Oct 30 19:31:43.108571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.108543    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:31:43.304352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304319    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-data\") pod \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") "
Oct 30 19:31:43.304554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304413    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-kube-api-access-926mt\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-kube-api-access-926mt\") pod \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") "
Oct 30 19:31:43.304554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304450    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-certs\") pod \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") "
Oct 30 19:31:43.304554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304538    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-token\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-token\") pod \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") "
Oct 30 19:31:43.304735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304611    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-data" pod "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" (UID: "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:43.304735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304618    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istiod-ca-cert\") pod \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") "
Oct 30 19:31:43.304735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304700    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-podinfo\") pod \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") "
Oct 30 19:31:43.304892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304831    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-credential-socket\") pod \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") "
Oct 30 19:31:43.304892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304845    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-certs" pod "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" (UID: "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:43.305008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304897    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-socket\") pod \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") "
Oct 30 19:31:43.305008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304935    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istiod-ca-cert" pod "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" (UID: "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:43.305008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.304965    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-envoy\") pod \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\" (UID: \"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78\") "
Oct 30 19:31:43.305168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.305135    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-credential-socket" pod "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" (UID: "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:43.305265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.305239    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-socket" pod "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" (UID: "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:43.306032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.306007    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:43.306171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.306037    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:43.306171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.306053    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:43.306171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.306084    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:43.306171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.306097    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:43.307038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.307002    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-kube-api-access-926mt" pod "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" (UID: "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78"). InnerVolumeSpecName "kube-api-access-926mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:43.307038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.307014    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-podinfo" pod "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" (UID: "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:31:43.307204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.307102    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-token" pod "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" (UID: "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:43.307204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.307187    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-envoy" pod "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" (UID: "bae35a03-f30c-43c2-83c6-cd5dcf3c1a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:43.407894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.407854    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:43.407894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.407886    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:43.407894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.407896    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:43.407894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.407905    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-926mt\" (UniqueName: \"kubernetes.io/projected/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78-kube-api-access-926mt\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:43.653897 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:31:43.653789    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:31:43.826218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.826182    6777 generic.go:356] "Generic (PLEG): container finished" podID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerID="4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af" exitCode=0
Oct 30 19:31:43.826397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.826228    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"
Oct 30 19:31:43.826397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.826262    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" event={"ID":"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78","Type":"ContainerDied","Data":"4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af"}
Oct 30 19:31:43.826397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.826322    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw" event={"ID":"bae35a03-f30c-43c2-83c6-cd5dcf3c1a78","Type":"ContainerDied","Data":"abf995ff3890c77adb4214d1207652d877324c0429e1ad7df945a6ea4b4017fa"}
Oct 30 19:31:43.826397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.826350    6777 scope.go:122] "RemoveContainer" containerID="4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af"
Oct 30 19:31:43.839429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.839268    6777 scope.go:122] "RemoveContainer" containerID="5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db"
Oct 30 19:31:43.855761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.855730    6777 scope.go:122] "RemoveContainer" containerID="4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af"
Oct 30 19:31:43.856166 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:31:43.856142    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af\": container with ID starting with 4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af not found: ID does not exist" containerID="4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af"
Oct 30 19:31:43.856256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.856185    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af"} err="failed to get container status \"4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af\": rpc error: code = NotFound desc = could not find container \"4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af\": container with ID starting with 4c7ec18bd64440b652eb864277d7ea7d8c40a0d2e22381f0ea85c33b414150af not found: ID does not exist"
Oct 30 19:31:43.856256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.856210    6777 scope.go:122] "RemoveContainer" containerID="5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db"
Oct 30 19:31:43.856558 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:31:43.856527    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db\": container with ID starting with 5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db not found: ID does not exist" containerID="5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db"
Oct 30 19:31:43.856666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.856568    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db"} err="failed to get container status \"5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db\": rpc error: code = NotFound desc = could not find container \"5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db\": container with ID starting with 5598a0ff249ce8e93125c8c9ee1bd3fb044530f1bf091319ea67feeb22d258db not found: ID does not exist"
Oct 30 19:31:43.861436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.861408    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"]
Oct 30 19:31:43.865494 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:43.865470    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4012a-openshift-default-7c884cc469-vrvmw"]
Oct 30 19:31:44.128133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:44.128096    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" path="/var/lib/kubelet/pods/bae35a03-f30c-43c2-83c6-cd5dcf3c1a78/volumes"
Oct 30 19:31:48.429343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:48.429310    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"]
Oct 30 19:31:48.466851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:48.466810    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"]
Oct 30 19:31:49.660415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.660376    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"]
Oct 30 19:31:49.661277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.661258    6777 cpu_manager.go:414] "containerMap: removing container" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy"
Oct 30 19:31:49.661353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.661278    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy"
Oct 30 19:31:49.661353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.661341    6777 cpu_manager.go:414] "containerMap: removing container" podUID="61f7e246-4210-4dd8-bd1a-775204614735" containerName="revoke-keys"
Oct 30 19:31:49.661353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.661347    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f7e246-4210-4dd8-bd1a-775204614735" containerName="revoke-keys"
Oct 30 19:31:49.661463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.661454    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy"
Oct 30 19:31:49.661502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.661468    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy"
Oct 30 19:31:49.661502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.661484    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f7e246-4210-4dd8-bd1a-775204614735" containerName="revoke-keys"
Oct 30 19:31:49.665295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.665275    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.675234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.675212    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-891d80-openshift-default-dockercfg-lk8xn\""
Oct 30 19:31:49.698413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.698377    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"]
Oct 30 19:31:49.800957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.800920    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07890996-9067-4a7d-934c-333cd93d4400-istiod-ca-cert\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.801171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.800968    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-envoy\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.801171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.801135    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-istio-token\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.801171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.801159    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-certs\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.801288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.801182    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-data\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.801353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.801332    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8kw\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-kube-api-access-dc8kw\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.801461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.801447    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-credential-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.801502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.801488    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07890996-9067-4a7d-934c-333cd93d4400-istio-podinfo\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.801615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.801600    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.902778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.902741    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07890996-9067-4a7d-934c-333cd93d4400-istiod-ca-cert\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.902932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.902788    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-envoy\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.902932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.902900    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-istio-token\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.902932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.902927    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-certs\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.903153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.902951    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-data\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.903153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.903049    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8kw\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-kube-api-access-dc8kw\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.903253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.903165    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-credential-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.903253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.903227    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07890996-9067-4a7d-934c-333cd93d4400-istio-podinfo\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.903382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.903312    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.903587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.903559    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07890996-9067-4a7d-934c-333cd93d4400-istiod-ca-cert\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.903951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.903909    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-certs\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.904097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.903956    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-data\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.904097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.903985    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-credential-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.904097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.903995    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.905815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.905790    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-envoy\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.906359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.906335    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07890996-9067-4a7d-934c-333cd93d4400-istio-podinfo\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.915608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.915546    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-istio-token\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.915608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.915578    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8kw\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-kube-api-access-dc8kw\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:49.981120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:49.981085    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:50.322921 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:50.322883    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07890996_9067_4a7d_934c_333cd93d4400.slice/crio-1b2918b4685abedbc21d791756abac90557dd3ed238ec3b125e961eaf6e8c4ff WatchSource:0}: Error finding container 1b2918b4685abedbc21d791756abac90557dd3ed238ec3b125e961eaf6e8c4ff: Status 404 returned error can't find the container with id 1b2918b4685abedbc21d791756abac90557dd3ed238ec3b125e961eaf6e8c4ff
Oct 30 19:31:50.325128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:50.324914    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"]
Oct 30 19:31:50.326798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:50.326759    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:50.326872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:50.326829    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:50.326872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:50.326858    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:31:50.909847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:50.909804    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" event={"ID":"07890996-9067-4a7d-934c-333cd93d4400","Type":"ContainerStarted","Data":"c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8"}
Oct 30 19:31:50.910389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:50.909876    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" event={"ID":"07890996-9067-4a7d-934c-333cd93d4400","Type":"ContainerStarted","Data":"1b2918b4685abedbc21d791756abac90557dd3ed238ec3b125e961eaf6e8c4ff"}
Oct 30 19:31:50.939053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:50.938999    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" podStartSLOduration=1.938985078 podStartE2EDuration="1.938985078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:50.93530519 +0000 UTC m=+1492.932220581" watchObservedRunningTime="2026-10-30 19:31:50.938985078 +0000 UTC m=+1492.935900457"
Oct 30 19:31:50.983084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:50.981269    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:51.808734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:51.808705    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:31:51.809500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:51.809464    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:31:51.990483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:51.990348    6777 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:31:51.990483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:51.990416    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:31:52.981836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:52.981799    6777 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:31:52.982076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:52.981863    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:31:53.434525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:53.434327    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf" podUID="56180ed3-f18c-4df7-a682-7cf948382c57" containerName="payload-processing" containerID="cri-o://1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b" gracePeriod=25
Oct 30 19:31:53.479236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:53.471326    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr" podUID="a5128a10-1b65-4461-be9b-3903830e5dbe" containerName="payload-pre-processing" containerID="cri-o://bcf0830678465aafef3bc99f5acf6bbca86ad3130f65dad8b8b9c001588f20f1" gracePeriod=25
Oct 30 19:31:53.864096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:53.848223    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:31:54.021141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.017188    6777 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:31:54.021141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.017257    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:31:54.045089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.035191    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56180ed3-f18c-4df7-a682-7cf948382c57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56180ed3-f18c-4df7-a682-7cf948382c57-plugins-config-volume\") pod \"56180ed3-f18c-4df7-a682-7cf948382c57\" (UID: \"56180ed3-f18c-4df7-a682-7cf948382c57\") "
Oct 30 19:31:54.045089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.035356    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56180ed3-f18c-4df7-a682-7cf948382c57-kube-api-access-s7tbc\" (UniqueName: \"kubernetes.io/projected/56180ed3-f18c-4df7-a682-7cf948382c57-kube-api-access-s7tbc\") pod \"56180ed3-f18c-4df7-a682-7cf948382c57\" (UID: \"56180ed3-f18c-4df7-a682-7cf948382c57\") "
Oct 30 19:31:54.076145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.075790    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56180ed3-f18c-4df7-a682-7cf948382c57-plugins-config-volume" pod "56180ed3-f18c-4df7-a682-7cf948382c57" (UID: "56180ed3-f18c-4df7-a682-7cf948382c57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:54.097083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.096732    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56180ed3-f18c-4df7-a682-7cf948382c57-kube-api-access-s7tbc" pod "56180ed3-f18c-4df7-a682-7cf948382c57" (UID: "56180ed3-f18c-4df7-a682-7cf948382c57"). InnerVolumeSpecName "kube-api-access-s7tbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:54.098752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.098716    6777 generic.go:356] "Generic (PLEG): container finished" podID="a5128a10-1b65-4461-be9b-3903830e5dbe" containerID="bcf0830678465aafef3bc99f5acf6bbca86ad3130f65dad8b8b9c001588f20f1" exitCode=0
Oct 30 19:31:54.099143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.098934    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr" event={"ID":"a5128a10-1b65-4461-be9b-3903830e5dbe","Type":"ContainerDied","Data":"bcf0830678465aafef3bc99f5acf6bbca86ad3130f65dad8b8b9c001588f20f1"}
Oct 30 19:31:54.115618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.102486    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:31:54.120150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.114697    6777 generic.go:356] "Generic (PLEG): container finished" podID="56180ed3-f18c-4df7-a682-7cf948382c57" containerID="1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b" exitCode=0
Oct 30 19:31:54.120720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.114810    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"
Oct 30 19:31:54.128144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.128107    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr" event={"ID":"a5128a10-1b65-4461-be9b-3903830e5dbe","Type":"ContainerDied","Data":"8a82fbb95a34bb773772532f87dd657bf6f48a64399170ddd3399e6a4433eb7c"}
Oct 30 19:31:54.131889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.130160    6777 scope.go:122] "RemoveContainer" containerID="bcf0830678465aafef3bc99f5acf6bbca86ad3130f65dad8b8b9c001588f20f1"
Oct 30 19:31:54.240723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.239980    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56180ed3-f18c-4df7-a682-7cf948382c57-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:54.240723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.240014    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7tbc\" (UniqueName: \"kubernetes.io/projected/56180ed3-f18c-4df7-a682-7cf948382c57-kube-api-access-s7tbc\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:54.261110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.259681    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf" event={"ID":"56180ed3-f18c-4df7-a682-7cf948382c57","Type":"ContainerDied","Data":"1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b"}
Oct 30 19:31:54.261110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.259725    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf" event={"ID":"56180ed3-f18c-4df7-a682-7cf948382c57","Type":"ContainerDied","Data":"3d56e09f8f6a7bd97491e3c245bd1e48e09916d966301324c487e60ec864bfa0"}
Oct 30 19:31:54.300099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.299994    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"]
Oct 30 19:31:54.303730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.303709    6777 scope.go:122] "RemoveContainer" containerID="1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b"
Oct 30 19:31:54.305997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.305971    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e6159-5fd5bcf88d-mmxjf"]
Oct 30 19:31:54.315923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.315900    6777 scope.go:122] "RemoveContainer" containerID="1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b"
Oct 30 19:31:54.316366 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:31:54.316347    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b\": container with ID starting with 1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b not found: ID does not exist" containerID="1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b"
Oct 30 19:31:54.316446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.316377    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b"} err="failed to get container status \"1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b\": rpc error: code = NotFound desc = could not find container \"1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b\": container with ID starting with 1fe735914c43143c0c9bdf76f585ffb5233d26a54ae4d590724e914124d8052b not found: ID does not exist"
Oct 30 19:31:54.340685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.340647    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5128a10-1b65-4461-be9b-3903830e5dbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5128a10-1b65-4461-be9b-3903830e5dbe-plugins-config-volume\") pod \"a5128a10-1b65-4461-be9b-3903830e5dbe\" (UID: \"a5128a10-1b65-4461-be9b-3903830e5dbe\") "
Oct 30 19:31:54.340904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.340857    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5128a10-1b65-4461-be9b-3903830e5dbe-kube-api-access-54t6q\" (UniqueName: \"kubernetes.io/projected/a5128a10-1b65-4461-be9b-3903830e5dbe-kube-api-access-54t6q\") pod \"a5128a10-1b65-4461-be9b-3903830e5dbe\" (UID: \"a5128a10-1b65-4461-be9b-3903830e5dbe\") "
Oct 30 19:31:54.341042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.341019    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5128a10-1b65-4461-be9b-3903830e5dbe-plugins-config-volume" pod "a5128a10-1b65-4461-be9b-3903830e5dbe" (UID: "a5128a10-1b65-4461-be9b-3903830e5dbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:54.341827 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.341802    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5128a10-1b65-4461-be9b-3903830e5dbe-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:54.343396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.343366    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5128a10-1b65-4461-be9b-3903830e5dbe-kube-api-access-54t6q" pod "a5128a10-1b65-4461-be9b-3903830e5dbe" (UID: "a5128a10-1b65-4461-be9b-3903830e5dbe"). InnerVolumeSpecName "kube-api-access-54t6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:54.443585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.443544    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54t6q\" (UniqueName: \"kubernetes.io/projected/a5128a10-1b65-4461-be9b-3903830e5dbe-kube-api-access-54t6q\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:54.983150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.983096    6777 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:31:54.983344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:54.983167    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:31:55.130241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:55.130208    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"
Oct 30 19:31:55.178786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:55.178727    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"]
Oct 30 19:31:55.182805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:55.182776    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e6159-647c947c95-tdxmr"]
Oct 30 19:31:55.982022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:55.981982    6777 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:31:55.982469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:55.982038    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:31:56.116459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:56.116423    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56180ed3-f18c-4df7-a682-7cf948382c57" path="/var/lib/kubelet/pods/56180ed3-f18c-4df7-a682-7cf948382c57/volumes"
Oct 30 19:31:56.116935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:56.116920    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5128a10-1b65-4461-be9b-3903830e5dbe" path="/var/lib/kubelet/pods/a5128a10-1b65-4461-be9b-3903830e5dbe/volumes"
Oct 30 19:31:56.981672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:56.981639    6777 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 19:31:56.981871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:56.981689    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 19:31:57.891090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.891022    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"]
Oct 30 19:31:57.891914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.891890    6777 cpu_manager.go:414] "containerMap: removing container" podUID="56180ed3-f18c-4df7-a682-7cf948382c57" containerName="payload-processing"
Oct 30 19:31:57.891914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.891909    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56180ed3-f18c-4df7-a682-7cf948382c57" containerName="payload-processing"
Oct 30 19:31:57.892186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.891932    6777 cpu_manager.go:414] "containerMap: removing container" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy"
Oct 30 19:31:57.892186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.891937    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae35a03-f30c-43c2-83c6-cd5dcf3c1a78" containerName="istio-proxy"
Oct 30 19:31:57.892186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.891971    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a5128a10-1b65-4461-be9b-3903830e5dbe" containerName="payload-pre-processing"
Oct 30 19:31:57.892186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.891977    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5128a10-1b65-4461-be9b-3903830e5dbe" containerName="payload-pre-processing"
Oct 30 19:31:57.892186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.892121    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="56180ed3-f18c-4df7-a682-7cf948382c57" containerName="payload-processing"
Oct 30 19:31:57.892186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.892141    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5128a10-1b65-4461-be9b-3903830e5dbe" containerName="payload-pre-processing"
Oct 30 19:31:57.897611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.897592    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:57.905426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.904803    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c24c01-serving-cert\""
Oct 30 19:31:57.909017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.908971    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"]
Oct 30 19:31:57.940229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.940193    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbnr2\" (UniqueName: \"kubernetes.io/projected/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-kube-api-access-qbnr2\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:57.940427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.940289    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-maas-api-tls\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:57.940506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.940478    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-metrics-certs\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:57.940576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.940538    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:57.987632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.987603    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:57.988107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.988084    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:57.988845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:57.988828    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:31:58.042878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.042832    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbnr2\" (UniqueName: \"kubernetes.io/projected/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-kube-api-access-qbnr2\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:58.043135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.042895    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-maas-api-tls\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:58.043135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.043073    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-metrics-certs\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:58.043135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.043101    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:58.043885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.043858    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:58.047092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.047051    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-maas-api-tls\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:58.047467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.047447    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-metrics-certs\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:58.059866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.059835    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbnr2\" (UniqueName: \"kubernetes.io/projected/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-kube-api-access-qbnr2\") pod \"maas-api-e2e-inf-a-c24c01-779664c894-lnfbs\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:58.222229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.222134    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:58.562443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.562408    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"]
Oct 30 19:31:58.563341 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:31:58.563314    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527730ba_24e1_4d87_9ffa_e0fcfb5cd2ce.slice/crio-a8931be8414ddd5482a9473ebc575a557533a2de340376b792f30e98f8354866 WatchSource:0}: Error finding container a8931be8414ddd5482a9473ebc575a557533a2de340376b792f30e98f8354866: Status 404 returned error can't find the container with id a8931be8414ddd5482a9473ebc575a557533a2de340376b792f30e98f8354866
Oct 30 19:31:58.564898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.564879    6777 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:31:58.765031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:58.765001    6777 kubelet.go:1689] "Image garbage collection succeeded"
Oct 30 19:31:59.211782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:59.211741    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs" event={"ID":"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce","Type":"ContainerStarted","Data":"8e099b5430764ea3eae3fc9dcc66308aee0c6af7a7ce23b6a9cc6d2b7bba77ae"}
Oct 30 19:31:59.212283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:59.211806    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs" event={"ID":"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce","Type":"ContainerStarted","Data":"a8931be8414ddd5482a9473ebc575a557533a2de340376b792f30e98f8354866"}
Oct 30 19:31:59.212283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:59.211931    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:31:59.234995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:59.234939    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs" podStartSLOduration=1.810342471 podStartE2EDuration="2.234924535s" totalImagesPullingTime="424.582064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:59.232272569 +0000 UTC m=+1501.229187956" watchObservedRunningTime="2026-10-30 19:31:59.234924535 +0000 UTC m=+1501.231839919"
Oct 30 19:31:59.603759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:59.603726    6777 scope.go:122] "RemoveContainer" containerID="4b04675224ee394fad7ee8e03d5bfb0e52702ecce5ed80d366e5766b120f0e13"
Oct 30 19:31:59.616252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:59.616229    6777 scope.go:122] "RemoveContainer" containerID="2e74b324a32afd276078429ea241619479816f5ef3d906075f40a7ea29b3c0e7"
Oct 30 19:31:59.626559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:31:59.626540    6777 scope.go:122] "RemoveContainer" containerID="99b4c0fa8866d3a0d8fd132bd8444c36eb1ced92d96c9d9c974cab2b17cb8f50"
Oct 30 19:32:02.391488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.391448    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"]
Oct 30 19:32:02.400165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.400135    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.417028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.407045    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"]
Oct 30 19:32:02.417028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.415798    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-69f3a9\"/\"kube-root-ca.crt\""
Oct 30 19:32:02.417620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.417535    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-69f3a9\"/\"openshift-service-ca.crt\""
Oct 30 19:32:02.417620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.417538    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-69f3a9\"/\"default-dockercfg-nj6hg\""
Oct 30 19:32:02.419331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.419307    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-69f3a9\"/\"test-model-69f3a9-kserve-self-signed-certs\""
Oct 30 19:32:02.530291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.530253    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tmp-dir\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.530445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.530307    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-model-cache\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.530557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.530532    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tls-certs\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.530704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.530686    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-dshm\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.530738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.530725    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4px\" (UniqueName: \"kubernetes.io/projected/f07f718a-19bb-4e4c-a4c4-f6a10b687225-kube-api-access-pj4px\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.530839 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.530828    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-home\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.634445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.634403    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tls-certs\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.634922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.634896    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-dshm\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.636350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.636325    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4px\" (UniqueName: \"kubernetes.io/projected/f07f718a-19bb-4e4c-a4c4-f6a10b687225-kube-api-access-pj4px\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.636562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.636546    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-home\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.644257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.644180    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tmp-dir\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.644257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.644238    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-model-cache\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.645012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.644986    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-dshm\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.645154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.640170    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-home\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.662648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.662611    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tmp-dir\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.664435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.664358    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-model-cache\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.691785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.665186    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tls-certs\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.691785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.665844    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4px\" (UniqueName: \"kubernetes.io/projected/f07f718a-19bb-4e4c-a4c4-f6a10b687225-kube-api-access-pj4px\") pod \"test-model-69f3a9-kserve-75cf4cc887-k8qnx\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") " pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:02.732700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:02.732577    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:03.072462 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:32:03.072430    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf07f718a_19bb_4e4c_a4c4_f6a10b687225.slice/crio-ee9a1f2bd6ead8bf98c036fe6ddd6d4024d72e5f749812f3171f07f2413a7126 WatchSource:0}: Error finding container ee9a1f2bd6ead8bf98c036fe6ddd6d4024d72e5f749812f3171f07f2413a7126: Status 404 returned error can't find the container with id ee9a1f2bd6ead8bf98c036fe6ddd6d4024d72e5f749812f3171f07f2413a7126
Oct 30 19:32:03.072994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:03.072969    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"]
Oct 30 19:32:03.246593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:03.246555    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx" event={"ID":"f07f718a-19bb-4e4c-a4c4-f6a10b687225","Type":"ContainerStarted","Data":"b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663"}
Oct 30 19:32:03.246759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:03.246599    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx" event={"ID":"f07f718a-19bb-4e4c-a4c4-f6a10b687225","Type":"ContainerStarted","Data":"ee9a1f2bd6ead8bf98c036fe6ddd6d4024d72e5f749812f3171f07f2413a7126"}
Oct 30 19:32:03.275930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:03.275879    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx" podStartSLOduration=1.275865434 podStartE2EDuration="1.275865434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:03.272994849 +0000 UTC m=+1505.269910233" watchObservedRunningTime="2026-10-30 19:32:03.275865434 +0000 UTC m=+1505.272780818"
Oct 30 19:32:03.732814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:03.732780    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:05.228469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:05.228437    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:32:12.733125 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:12.733088    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:12.745791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:12.745751    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:13.706589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:13.706522    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:32:13.949082 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:32:13.941744    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:32:22.568217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:22.568147    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"]
Oct 30 19:32:22.574584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:22.574551    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e6159-6fmmf"]
Oct 30 19:32:24.120789 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:24.120754    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f7e246-4210-4dd8-bd1a-775204614735" path="/var/lib/kubelet/pods/61f7e246-4210-4dd8-bd1a-775204614735/volumes"
Oct 30 19:32:25.124542 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.123333    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"]
Oct 30 19:32:25.267440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.264844    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.287815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.285898    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"]
Oct 30 19:32:25.303844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.302945    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-76117c-openshift-default-dockercfg-4nkjx\""
Oct 30 19:32:25.354119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.354080    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15081ca-227c-428f-ad9e-a5bac902df6e-istiod-ca-cert\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.354308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.354167    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-podinfo\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.354308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.354241    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-credential-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.354424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.354379    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv59z\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-kube-api-access-cv59z\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.354570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.354550    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-token\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.354614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.354595    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.354649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.354622    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-certs\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.354802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.354780    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-envoy\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.354854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.354840    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-data\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.456626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.456575    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-podinfo\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.456799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.456699    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-credential-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.456858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.456820    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv59z\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-kube-api-access-cv59z\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.456971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.456954    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-token\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.457049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.456997    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.457049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.457025    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-certs\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.457273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.457252    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-credential-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.457502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.457444    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-certs\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.457502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.457484    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.457672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.457560    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-envoy\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.457672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.457613    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-data\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.457774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.457736    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15081ca-227c-428f-ad9e-a5bac902df6e-istiod-ca-cert\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.458573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.458545    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15081ca-227c-428f-ad9e-a5bac902df6e-istiod-ca-cert\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.459943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.459901    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-data\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.461749 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.461724    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-envoy\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.462885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.462867    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-podinfo\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.471775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.471746    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-token\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.473937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.473910    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv59z\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-kube-api-access-cv59z\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.639831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.639474    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:25.640752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.640725    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"]
Oct 30 19:32:25.645366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.645343    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.665800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.665766    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.665956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.665909    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3692e68-9f66-4f51-b682-3d91a22b187e-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.666039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.666007    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.666164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.666037    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.666164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.666089    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.666288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.666205    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.666384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.666367    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.666483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.666467    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2q4\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-kube-api-access-gx2q4\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.666574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.666560    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.668184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.668156    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"]
Oct 30 19:32:25.769345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.768757    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3692e68-9f66-4f51-b682-3d91a22b187e-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.769345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.768870    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.769345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.768900    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.769345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.768935    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.769345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.769024    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.769345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.769143    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.769345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.769226    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2q4\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-kube-api-access-gx2q4\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.769345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.769297    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.772097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.771203    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.772097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.771581    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3692e68-9f66-4f51-b682-3d91a22b187e-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.772097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.771892    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.772097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.770815    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.772097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.771974    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.772558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.772343    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.773712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.773679    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.775191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.775167    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.788872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.788844    6777 generic.go:356] "Generic (PLEG): container finished" podID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerID="14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3" exitCode=0
Oct 30 19:32:25.788994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.788921    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" event={"ID":"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799","Type":"ContainerDied","Data":"14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3"}
Oct 30 19:32:25.789085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.789019    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2q4\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-kube-api-access-gx2q4\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.789556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.789529    6777 scope.go:122] "RemoveContainer" containerID="14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3"
Oct 30 19:32:25.789672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.789591    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:25.790871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.790844    6777 generic.go:356] "Generic (PLEG): container finished" podID="11d65393-28cb-433f-9157-0ad2f5e90083" containerID="8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496" exitCode=0
Oct 30 19:32:25.790987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.790885    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" event={"ID":"11d65393-28cb-433f-9157-0ad2f5e90083","Type":"ContainerDied","Data":"8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496"}
Oct 30 19:32:25.791366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.791349    6777 scope.go:122] "RemoveContainer" containerID="8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496"
Oct 30 19:32:25.791649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.791622    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:25.791732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.791690    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:25.791732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.791725    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:25.793263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.793234    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:25.793378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.793297    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:25.793378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.793329    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:25.964262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.964218    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"]
Oct 30 19:32:25.982635 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.982589    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:25.982779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.982689    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:25.982779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:25.982734    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:26.017118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.014790    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:26.332716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.332685    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"]
Oct 30 19:32:26.335621 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:32:26.335585    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3692e68_9f66_4f51_b682_3d91a22b187e.slice/crio-322e5853fd6c17ded07a00f4e22690d01378e4a9618bbda0eb821e7e2b9444ad WatchSource:0}: Error finding container 322e5853fd6c17ded07a00f4e22690d01378e4a9618bbda0eb821e7e2b9444ad: Status 404 returned error can't find the container with id 322e5853fd6c17ded07a00f4e22690d01378e4a9618bbda0eb821e7e2b9444ad
Oct 30 19:32:26.343369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.343327    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:26.343496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.343419    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:26.343496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.343461    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:26.802666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.802621    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" event={"ID":"a15081ca-227c-428f-ad9e-a5bac902df6e","Type":"ContainerStarted","Data":"4b7738d58fe2859a719d9855578fbd91896fe6bba476c7b9f1fa63580a81f858"}
Oct 30 19:32:26.802882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.802698    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" event={"ID":"a15081ca-227c-428f-ad9e-a5bac902df6e","Type":"ContainerStarted","Data":"4bc83f4495c86339b98129f9b90c56c9d6a9c3fb674417c157e1f57c85005530"}
Oct 30 19:32:26.805227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.805200    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" event={"ID":"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799","Type":"ContainerStarted","Data":"7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5"}
Oct 30 19:32:26.807621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.807590    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" event={"ID":"11d65393-28cb-433f-9157-0ad2f5e90083","Type":"ContainerStarted","Data":"c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69"}
Oct 30 19:32:26.810528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.810501    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" event={"ID":"f3692e68-9f66-4f51-b682-3d91a22b187e","Type":"ContainerStarted","Data":"b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a"}
Oct 30 19:32:26.810675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.810551    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" event={"ID":"f3692e68-9f66-4f51-b682-3d91a22b187e","Type":"ContainerStarted","Data":"322e5853fd6c17ded07a00f4e22690d01378e4a9618bbda0eb821e7e2b9444ad"}
Oct 30 19:32:26.845613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.845579    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:32:26.846616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.846590    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:32:26.846757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.846643    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 19:32:26.897456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.897415    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:32:26.897693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.897670    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:32:26.897752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.897728    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 19:32:26.911847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:26.911791    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" podStartSLOduration=1.91177711 podStartE2EDuration="1.91177711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:26.909686446 +0000 UTC m=+1528.906601833" watchObservedRunningTime="2026-10-30 19:32:26.91177711 +0000 UTC m=+1528.908692495"
Oct 30 19:32:27.010157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.010082    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podStartSLOduration=2.010046427 podStartE2EDuration="2.010046427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:27.008642709 +0000 UTC m=+1529.005558094" watchObservedRunningTime="2026-10-30 19:32:27.010046427 +0000 UTC m=+1529.006961813"
Oct 30 19:32:27.016282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.016245    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:27.018005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.017974    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:32:27.018172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.018035    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 19:32:27.640544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.640509    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:27.646781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.646755    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:27.848090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.845174    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:32:27.848090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.845238    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 19:32:27.922144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.908932    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:32:27.922144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.908988    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 19:32:27.922144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.919643    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:27.922144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:27.921957    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:32:28.017082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:28.016721    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:32:28.017082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:28.016779    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 19:32:28.846648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:28.846609    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:32:28.847077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:28.846764    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 19:32:28.919097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:28.917770    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:32:28.919097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:28.917833    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 19:32:29.035252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:29.035209    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:32:29.035446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:29.035304    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 19:32:29.845311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:29.845278    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:32:29.845648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:29.845625    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 19:32:29.918085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:29.915855    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:32:29.918085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:29.915965    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 19:32:30.029894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:30.024360    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:32:30.029894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:30.024420    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 19:32:30.666191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:30.666155    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:32:30.933139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:30.928165    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:32:30.933139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:30.928541    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:32:30.933139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:30.928594    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 19:32:31.041221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.021110    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 19:32:31.041221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.031359    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:32:31.041221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.031416    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 19:32:31.284165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.284133    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"]
Oct 30 19:32:31.686143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.655247    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:31.686143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.660354    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"]
Oct 30 19:32:31.686143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.667458    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-76117c\""
Oct 30 19:32:31.686143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.670831    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-76117c-dockercfg-thszs\""
Oct 30 19:32:31.724304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.706953    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"]
Oct 30 19:32:31.724304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.719278    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"]
Oct 30 19:32:31.724304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.719379    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:31.820222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.806737    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdmbw\" (UniqueName: \"kubernetes.io/projected/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-kube-api-access-pdmbw\") pod \"payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd\" (UID: \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:31.820222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.806833    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd\" (UID: \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:31.820222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.818908    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf" podUID="76810fef-bb01-431d-b4a9-faf73e8f25ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:31.904088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.897334    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:32:31.904088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.897394    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 19:32:31.936081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.934878    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:31.949369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.948883    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:32.018082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.998812    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmbw\" (UniqueName: \"kubernetes.io/projected/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-kube-api-access-pdmbw\") pod \"payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd\" (UID: \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:32.018082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.998944    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd\" (UID: \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:32.018082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.999244    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5\" (UID: \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:32.018082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:31.999480    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwcww\" (UniqueName: \"kubernetes.io/projected/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-kube-api-access-wwcww\") pod \"payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5\" (UID: \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:32.018082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.000857    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd\" (UID: \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:32.068180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.038599    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:32:32.068180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.049753    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:32:32.068180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.049810    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 19:32:32.068180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.054549    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:32:32.068180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.054594    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 19:32:32.106090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.094380    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmbw\" (UniqueName: \"kubernetes.io/projected/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-kube-api-access-pdmbw\") pod \"payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd\" (UID: \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:32.126085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.122161    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwcww\" (UniqueName: \"kubernetes.io/projected/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-kube-api-access-wwcww\") pod \"payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5\" (UID: \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:32.126085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.122550    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5\" (UID: \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:32.126085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.123838    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5\" (UID: \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:32.223179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.222888    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwcww\" (UniqueName: \"kubernetes.io/projected/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-kube-api-access-wwcww\") pod \"payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5\" (UID: \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:32.225668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.225633    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:32.286791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.256577    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:32.317739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.317174    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:32.845080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.845031    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:32:32.845407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.845374    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 19:32:32.931086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.930170    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:32:32.931086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:32.930239    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 19:32:33.035082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:33.033816    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:32:33.035082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:33.033875    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 19:32:33.447329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:33.447297    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:32:33.593819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:33.593779    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"]
Oct 30 19:32:33.741088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:33.733183    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5" event={"ID":"31a6c3c2-002f-45db-a84b-31bc40cb6dd5","Type":"ContainerStarted","Data":"35bb4fc092e455dde096acb6efd404bfe95c4641a199b2cbe6401d6cbcd3fdd2"}
Oct 30 19:32:33.841131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:33.834216    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"]
Oct 30 19:32:33.899592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:33.899555    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:32:33.899789 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:33.899617    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 19:32:34.017481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.017432    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:32:34.022758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.021786    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:32:34.022758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.022157    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:32:34.022758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.022207    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 19:32:34.745689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.745652    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd" event={"ID":"f78e5730-0b9f-45b1-8e5a-16f1d96e3228","Type":"ContainerStarted","Data":"066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497"}
Oct 30 19:32:34.746181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.745717    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd" event={"ID":"f78e5730-0b9f-45b1-8e5a-16f1d96e3228","Type":"ContainerStarted","Data":"a4bc135e66a0f4badb15658a138e00c0ea9f8023450b36c6d51c1c61d26545f8"}
Oct 30 19:32:34.746181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.745783    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:34.747783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.747748    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5" event={"ID":"31a6c3c2-002f-45db-a84b-31bc40cb6dd5","Type":"ContainerStarted","Data":"2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032"}
Oct 30 19:32:34.747944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.747844    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:34.773693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.773634    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd" podStartSLOduration=3.773617772 podStartE2EDuration="3.773617772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:34.770372903 +0000 UTC m=+1536.767288308" watchObservedRunningTime="2026-10-30 19:32:34.773617772 +0000 UTC m=+1536.770533157"
Oct 30 19:32:34.815023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.814952    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5" podStartSLOduration=3.814934379 podStartE2EDuration="3.814934379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:34.801093474 +0000 UTC m=+1536.798008860" watchObservedRunningTime="2026-10-30 19:32:34.814934379 +0000 UTC m=+1536.811849766"
Oct 30 19:32:34.905385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.905346    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:32:34.906874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:34.906848    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:32:35.040209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.039430    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:35.041198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.041149    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:35.042646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.042626    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:32:35.325366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.325267    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"]
Oct 30 19:32:35.340536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.340508    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.351034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.351002    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"]
Oct 30 19:32:35.351680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.351655    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f2dbd9-openshift-default-dockercfg-hl6bt\""
Oct 30 19:32:35.494081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.486546    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-podinfo\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.494081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.486662    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xfzx\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-kube-api-access-2xfzx\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.494081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.486724    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13df2c0-4690-4548-bf78-a4205a6cae13-istiod-ca-cert\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.494081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.486775    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-certs\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.494081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.486805    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-credential-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.494081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.486830    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-data\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.494081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.486867    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.494081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.487178    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-token\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.494081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.487327    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-envoy\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.608802    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-token\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.608932    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-envoy\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.609108    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-podinfo\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.609172    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xfzx\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-kube-api-access-2xfzx\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.609210    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13df2c0-4690-4548-bf78-a4205a6cae13-istiod-ca-cert\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.609243    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-certs\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.609273    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-credential-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.609296    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-data\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.609327    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.613222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.609917    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.614619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.614589    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-credential-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.615014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.614984    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-certs\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.620269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.620242    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-data\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.623098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.621410    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13df2c0-4690-4548-bf78-a4205a6cae13-istiod-ca-cert\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.643885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.643850    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-podinfo\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.667514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.667475    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xfzx\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-kube-api-access-2xfzx\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.668877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.668849    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-envoy\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.669951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.669878    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-token\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:35.964207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:35.964131    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:36.289049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:36.289005    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"]
Oct 30 19:32:36.317488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:36.317451    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:36.317622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:36.317533    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:36.317622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:36.317567    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:36.774577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:36.774529    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" event={"ID":"c13df2c0-4690-4548-bf78-a4205a6cae13","Type":"ContainerStarted","Data":"2fdc81f7cc88d01643eda59c2118cff4ba5f8101223e33927ec67bc91ddfc760"}
Oct 30 19:32:36.774828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:36.774630    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" event={"ID":"c13df2c0-4690-4548-bf78-a4205a6cae13","Type":"ContainerStarted","Data":"9fb8ceecf5145e81b489b41811b308da97ce9fada11b8abddc7358e300272d7f"}
Oct 30 19:32:36.810563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:36.810505    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podStartSLOduration=1.810489876 podStartE2EDuration="1.810489876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:36.805973572 +0000 UTC m=+1538.802888958" watchObservedRunningTime="2026-10-30 19:32:36.810489876 +0000 UTC m=+1538.807405301"
Oct 30 19:32:36.968748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:36.964275    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:37.971170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:37.971139    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:38.611405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:38.611370    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:32:38.843080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:38.841535    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:38.860080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:38.848635    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:32:43.647474 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:32:43.647432    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:32:45.773896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:45.773862    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:32:45.774374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:45.773954    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:32:46.854528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.854495    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"]
Oct 30 19:32:46.862843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.862810    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:46.874921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.874890    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e3586f\"/\"auth-test-model-e3586f-kserve-self-signed-certs\""
Oct 30 19:32:46.875158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.874980    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e3586f\"/\"default-dockercfg-bpqwz\""
Oct 30 19:32:46.880401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.880362    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e3586f\"/\"openshift-service-ca.crt\""
Oct 30 19:32:46.880570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.880456    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e3586f\"/\"kube-root-ca.crt\""
Oct 30 19:32:46.895658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.895625    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"]
Oct 30 19:32:46.913808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.913761    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-home\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:46.914017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.913834    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-tmp-dir\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:46.914017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.913877    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da1619c-e802-47a9-abff-7406f07cf890-tls-certs\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:46.914163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.914135    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-model-cache\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:46.914327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.914306    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvk6q\" (UniqueName: \"kubernetes.io/projected/0da1619c-e802-47a9-abff-7406f07cf890-kube-api-access-mvk6q\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:46.914438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:46.914358    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-dshm\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.016489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.016461    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-home\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.016639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.016496    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-tmp-dir\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.016639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.016531    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da1619c-e802-47a9-abff-7406f07cf890-tls-certs\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.016720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.016679    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-model-cache\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.016814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.016795    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvk6q\" (UniqueName: \"kubernetes.io/projected/0da1619c-e802-47a9-abff-7406f07cf890-kube-api-access-mvk6q\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.016859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.016836    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-dshm\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.016927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.016908    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-home\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.017011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.016966    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-tmp-dir\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.017086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.017012    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-model-cache\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.018953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.018931    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-dshm\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.019293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.019270    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da1619c-e802-47a9-abff-7406f07cf890-tls-certs\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.030267 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.030235    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvk6q\" (UniqueName: \"kubernetes.io/projected/0da1619c-e802-47a9-abff-7406f07cf890-kube-api-access-mvk6q\") pod \"auth-test-model-e3586f-kserve-84fdf76b5-7rsx5\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.204901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.204740    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:47.548513 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:32:47.548478    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da1619c_e802_47a9_abff_7406f07cf890.slice/crio-64ee45e22a05447fdad02d896f9e88a3c264ac5a74036f856b36b4f3986b0635 WatchSource:0}: Error finding container 64ee45e22a05447fdad02d896f9e88a3c264ac5a74036f856b36b4f3986b0635: Status 404 returned error can't find the container with id 64ee45e22a05447fdad02d896f9e88a3c264ac5a74036f856b36b4f3986b0635
Oct 30 19:32:47.548755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:47.548735    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"]
Oct 30 19:32:48.144602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:48.144565    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5" event={"ID":"0da1619c-e802-47a9-abff-7406f07cf890","Type":"ContainerStarted","Data":"74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c"}
Oct 30 19:32:48.144602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:48.144612    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5" event={"ID":"0da1619c-e802-47a9-abff-7406f07cf890","Type":"ContainerStarted","Data":"64ee45e22a05447fdad02d896f9e88a3c264ac5a74036f856b36b4f3986b0635"}
Oct 30 19:32:48.166631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:48.166583    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5" podStartSLOduration=2.166568258 podStartE2EDuration="2.166568258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:48.165234269 +0000 UTC m=+1550.162149653" watchObservedRunningTime="2026-10-30 19:32:48.166568258 +0000 UTC m=+1550.163483642"
Oct 30 19:32:48.204918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:48.204887    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:52.181333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:52.181238    6777 generic.go:356] "Generic (PLEG): container finished" podID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerID="71ef753f4bae2c2cc33b7646002cf92a65e1fc90d3666a0c8a34fc20876231dd" exitCode=0
Oct 30 19:32:52.181333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:52.181306    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" event={"ID":"9dcc116f-353d-4ba8-836b-ca158e68bac4","Type":"ContainerDied","Data":"71ef753f4bae2c2cc33b7646002cf92a65e1fc90d3666a0c8a34fc20876231dd"}
Oct 30 19:32:52.181807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:52.181780    6777 scope.go:122] "RemoveContainer" containerID="71ef753f4bae2c2cc33b7646002cf92a65e1fc90d3666a0c8a34fc20876231dd"
Oct 30 19:32:52.184228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:52.184197    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:52.184333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:52.184286    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:52.184379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:52.184328    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:32:53.192823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:53.192785    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" event={"ID":"9dcc116f-353d-4ba8-836b-ca158e68bac4","Type":"ContainerStarted","Data":"d6f91ee3d8a5dfbba58c0fb8deceec31a440fe108848aae3e0e5281478772753"}
Oct 30 19:32:53.405468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:53.405436    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:32:53.406736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:53.406701    6777 patch_prober.go:29] interesting pod/e2e-inf-b-c86696-openshift-default-859496475c-srmqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 19:32:53.406893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:53.406767    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 19:32:54.412241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:54.412198    6777 patch_prober.go:29] interesting pod/e2e-inf-b-c86696-openshift-default-859496475c-srmqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 19:32:54.412865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:54.412827    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 19:32:55.045012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:55.044974    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:32:55.405331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:55.405238    6777 patch_prober.go:29] interesting pod/e2e-inf-b-c86696-openshift-default-859496475c-srmqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 19:32:55.405331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:55.405309    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 19:32:56.405425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:56.405389    6777 patch_prober.go:29] interesting pod/e2e-inf-b-c86696-openshift-default-859496475c-srmqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 19:32:56.405838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:56.405444    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 19:32:57.205605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:57.205570    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:57.218703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:57.218672    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:57.243161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:57.243132    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:32:57.405797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:57.405750    6777 patch_prober.go:29] interesting pod/e2e-inf-b-c86696-openshift-default-859496475c-srmqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 19:32:57.406319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:57.405821    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 19:32:58.405459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:58.405424    6777 patch_prober.go:29] interesting pod/e2e-inf-b-c86696-openshift-default-859496475c-srmqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 19:32:58.405677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:58.405491    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 19:32:59.410524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:59.410494    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:32:59.411676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:32:59.411654    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:33:07.722034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.721988    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"]
Oct 30 19:33:07.728538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.728515    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:07.735090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.735051    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:33:07.735248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.735169    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:33:07.739902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.739874    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"]
Oct 30 19:33:07.771625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.771590    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv57s\" (UniqueName: \"kubernetes.io/projected/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-kube-api-access-zv57s\") pod \"maas-api-revoke-sub-e43e37f7-68mtl\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:07.771941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.771910    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e43e37f7-68mtl\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:07.772114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.772097    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e43e37f7-68mtl\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:07.874114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.874077    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e43e37f7-68mtl\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:07.874296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.874188    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e43e37f7-68mtl\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:07.874296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.874289    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv57s\" (UniqueName: \"kubernetes.io/projected/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-kube-api-access-zv57s\") pod \"maas-api-revoke-sub-e43e37f7-68mtl\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:07.874799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.874774    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e43e37f7-68mtl\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:07.876748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.876722    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e43e37f7-68mtl\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:07.886870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:07.886843    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv57s\" (UniqueName: \"kubernetes.io/projected/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-kube-api-access-zv57s\") pod \"maas-api-revoke-sub-e43e37f7-68mtl\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:08.055945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:08.055910    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:08.410142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:08.400362    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"]
Oct 30 19:33:08.425660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:08.425632    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl" event={"ID":"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10","Type":"ContainerStarted","Data":"e6dabfbe7dc858a01acaa2581ef50c8d1d66bdb99417b5c3edddf554d4bda1d3"}
Oct 30 19:33:09.435785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:09.435746    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl" event={"ID":"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10","Type":"ContainerStarted","Data":"ee37b735f2b9562912f497b41eace9a362e7155f37b4a4422186ca70d98c6984"}
Oct 30 19:33:09.466156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:09.466101    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl" podStartSLOduration=2.46608699 podStartE2EDuration="2.46608699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:09.461460758 +0000 UTC m=+1571.458376142" watchObservedRunningTime="2026-10-30 19:33:09.46608699 +0000 UTC m=+1571.463002365"
Oct 30 19:33:10.446292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:10.446252    6777 generic.go:356] "Generic (PLEG): container finished" podID="7f682adc-fe85-4781-9b5d-f0a9f9f6ba10" containerID="ee37b735f2b9562912f497b41eace9a362e7155f37b4a4422186ca70d98c6984" exitCode=0
Oct 30 19:33:10.446739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:10.446349    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl" event={"ID":"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10","Type":"ContainerDied","Data":"ee37b735f2b9562912f497b41eace9a362e7155f37b4a4422186ca70d98c6984"}
Oct 30 19:33:11.674247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.672736    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:11.778247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.778154    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-xks-ca\") pod \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") "
Oct 30 19:33:11.778432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.778286    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-kube-api-access-zv57s\" (UniqueName: \"kubernetes.io/projected/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-kube-api-access-zv57s\") pod \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") "
Oct 30 19:33:11.778624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.778487    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-openshift-ca\") pod \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\" (UID: \"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10\") "
Oct 30 19:33:11.779098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.778987    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-openshift-ca" pod "7f682adc-fe85-4781-9b5d-f0a9f9f6ba10" (UID: "7f682adc-fe85-4781-9b5d-f0a9f9f6ba10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:11.779467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.779447    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:11.781261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.781230    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-kube-api-access-zv57s" pod "7f682adc-fe85-4781-9b5d-f0a9f9f6ba10" (UID: "7f682adc-fe85-4781-9b5d-f0a9f9f6ba10"). InnerVolumeSpecName "kube-api-access-zv57s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:11.781843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.781815    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-xks-ca" pod "7f682adc-fe85-4781-9b5d-f0a9f9f6ba10" (UID: "7f682adc-fe85-4781-9b5d-f0a9f9f6ba10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:11.881718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.881682    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:11.881718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:11.881713    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv57s\" (UniqueName: \"kubernetes.io/projected/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10-kube-api-access-zv57s\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:12.548765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:12.548702    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl" event={"ID":"7f682adc-fe85-4781-9b5d-f0a9f9f6ba10","Type":"ContainerDied","Data":"e6dabfbe7dc858a01acaa2581ef50c8d1d66bdb99417b5c3edddf554d4bda1d3"}
Oct 30 19:33:12.548946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:12.548827    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6dabfbe7dc858a01acaa2581ef50c8d1d66bdb99417b5c3edddf554d4bda1d3"
Oct 30 19:33:12.548946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:12.548924    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"
Oct 30 19:33:13.643781 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:33:13.643745    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:33:28.923595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:28.922372    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"]
Oct 30 19:33:29.488273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.488239    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"]
Oct 30 19:33:29.489613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.489592    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7f682adc-fe85-4781-9b5d-f0a9f9f6ba10" containerName="revoke-keys"
Oct 30 19:33:29.489700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.489615    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f682adc-fe85-4781-9b5d-f0a9f9f6ba10" containerName="revoke-keys"
Oct 30 19:33:29.489864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.489850    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f682adc-fe85-4781-9b5d-f0a9f9f6ba10" containerName="revoke-keys"
Oct 30 19:33:29.495760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.494911    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.501750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.501718    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:33:29.502125 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.502100    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:33:29.504452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.504423    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"]
Oct 30 19:33:29.562188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.562136    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-597cded3-pcwjt\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.562384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.562199    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr6l7\" (UniqueName: \"kubernetes.io/projected/c6d096e9-4314-49e3-a6bb-faa6a49e5563-kube-api-access-rr6l7\") pod \"maas-api-revoke-sub-597cded3-pcwjt\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.562488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.562460    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-597cded3-pcwjt\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.664359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.664314    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-597cded3-pcwjt\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.664359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.664363    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr6l7\" (UniqueName: \"kubernetes.io/projected/c6d096e9-4314-49e3-a6bb-faa6a49e5563-kube-api-access-rr6l7\") pod \"maas-api-revoke-sub-597cded3-pcwjt\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.664622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.664466    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-597cded3-pcwjt\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.665276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.665232    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-597cded3-pcwjt\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.667029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.667005    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-597cded3-pcwjt\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.679144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.679103    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr6l7\" (UniqueName: \"kubernetes.io/projected/c6d096e9-4314-49e3-a6bb-faa6a49e5563-kube-api-access-rr6l7\") pod \"maas-api-revoke-sub-597cded3-pcwjt\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:29.822396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:29.822359    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:30.119173 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:33:30.119141    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d096e9_4314_49e3_a6bb_faa6a49e5563.slice/crio-dc8f563538fa9233f83e673cf55a66bf475b25f1cadc1003195066b5442cf206 WatchSource:0}: Error finding container dc8f563538fa9233f83e673cf55a66bf475b25f1cadc1003195066b5442cf206: Status 404 returned error can't find the container with id dc8f563538fa9233f83e673cf55a66bf475b25f1cadc1003195066b5442cf206
Oct 30 19:33:30.124373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:30.124284    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"]
Oct 30 19:33:30.865976 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:30.865936    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt" event={"ID":"c6d096e9-4314-49e3-a6bb-faa6a49e5563","Type":"ContainerStarted","Data":"0e9665f82a6d16a059927973707991a0e19795d675c3826e4eba1d37bb648731"}
Oct 30 19:33:30.866262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:30.866000    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt" event={"ID":"c6d096e9-4314-49e3-a6bb-faa6a49e5563","Type":"ContainerStarted","Data":"dc8f563538fa9233f83e673cf55a66bf475b25f1cadc1003195066b5442cf206"}
Oct 30 19:33:30.889933 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:30.889861    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt" podStartSLOduration=1.889842217 podStartE2EDuration="1.889842217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:30.888169704 +0000 UTC m=+1592.885085088" watchObservedRunningTime="2026-10-30 19:33:30.889842217 +0000 UTC m=+1592.886757603"
Oct 30 19:33:31.878149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:31.878106    6777 generic.go:356] "Generic (PLEG): container finished" podID="c6d096e9-4314-49e3-a6bb-faa6a49e5563" containerID="0e9665f82a6d16a059927973707991a0e19795d675c3826e4eba1d37bb648731" exitCode=0
Oct 30 19:33:31.878595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:31.878188    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt" event={"ID":"c6d096e9-4314-49e3-a6bb-faa6a49e5563","Type":"ContainerDied","Data":"0e9665f82a6d16a059927973707991a0e19795d675c3826e4eba1d37bb648731"}
Oct 30 19:33:33.437835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.437796    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:33.568175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.568134    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d096e9-4314-49e3-a6bb-faa6a49e5563-kube-api-access-rr6l7\" (UniqueName: \"kubernetes.io/projected/c6d096e9-4314-49e3-a6bb-faa6a49e5563-kube-api-access-rr6l7\") pod \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") "
Oct 30 19:33:33.568371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.568276    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-xks-ca\") pod \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") "
Oct 30 19:33:33.568427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.568404    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-openshift-ca\") pod \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\" (UID: \"c6d096e9-4314-49e3-a6bb-faa6a49e5563\") "
Oct 30 19:33:33.568777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.568748    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-openshift-ca" pod "c6d096e9-4314-49e3-a6bb-faa6a49e5563" (UID: "c6d096e9-4314-49e3-a6bb-faa6a49e5563"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:33.569160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.569143    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:33.640556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.640510    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-xks-ca" pod "c6d096e9-4314-49e3-a6bb-faa6a49e5563" (UID: "c6d096e9-4314-49e3-a6bb-faa6a49e5563"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:33.640715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.640567    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d096e9-4314-49e3-a6bb-faa6a49e5563-kube-api-access-rr6l7" pod "c6d096e9-4314-49e3-a6bb-faa6a49e5563" (UID: "c6d096e9-4314-49e3-a6bb-faa6a49e5563"). InnerVolumeSpecName "kube-api-access-rr6l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:33.670889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.670850    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr6l7\" (UniqueName: \"kubernetes.io/projected/c6d096e9-4314-49e3-a6bb-faa6a49e5563-kube-api-access-rr6l7\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:33.670889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.670889    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d096e9-4314-49e3-a6bb-faa6a49e5563-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:33.904118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.904017    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"
Oct 30 19:33:33.904402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.904033    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt" event={"ID":"c6d096e9-4314-49e3-a6bb-faa6a49e5563","Type":"ContainerDied","Data":"dc8f563538fa9233f83e673cf55a66bf475b25f1cadc1003195066b5442cf206"}
Oct 30 19:33:33.904515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:33.904470    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc8f563538fa9233f83e673cf55a66bf475b25f1cadc1003195066b5442cf206"
Oct 30 19:33:36.041081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:36.040950    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"]
Oct 30 19:33:36.042000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:36.041966    6777 generic.go:356] "Generic (PLEG): container finished" podID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerID="4b7738d58fe2859a719d9855578fbd91896fe6bba476c7b9f1fa63580a81f858" exitCode=0
Oct 30 19:33:36.042377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:36.042349    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" event={"ID":"a15081ca-227c-428f-ad9e-a5bac902df6e","Type":"ContainerDied","Data":"4b7738d58fe2859a719d9855578fbd91896fe6bba476c7b9f1fa63580a81f858"}
Oct 30 19:33:36.043161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:36.043136    6777 scope.go:122] "RemoveContainer" containerID="4b7738d58fe2859a719d9855578fbd91896fe6bba476c7b9f1fa63580a81f858"
Oct 30 19:33:36.047240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:36.047180    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:33:36.047385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:36.047360    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:33:36.047515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:36.047469    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:33:36.050969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:36.050890    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5f10ff-grnps"]
Oct 30 19:33:36.129542 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:36.128464    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02f4f5e-8d7e-412e-9b05-dc2893af31d1" path="/var/lib/kubelet/pods/b02f4f5e-8d7e-412e-9b05-dc2893af31d1/volumes"
Oct 30 19:33:37.062522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:37.062329    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" event={"ID":"a15081ca-227c-428f-ad9e-a5bac902df6e","Type":"ContainerStarted","Data":"33b672524a8ddddfaa5ffb07dc63af46321942c048eaca53e61465008f3a472b"}
Oct 30 19:33:37.641215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:37.641178    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:33:37.641734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:37.641710    6777 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:33:37.641794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:37.641775    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 19:33:38.641378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:38.641335    6777 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:33:38.641809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:38.641397    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 19:33:39.643429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:39.641941    6777 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:33:39.643429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:39.642010    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 19:33:40.640683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:40.640639    6777 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:33:40.640906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:40.640703    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 19:33:41.640689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:41.640652    6777 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:33:41.641154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:41.640713    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 19:33:42.641008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:42.640970    6777 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:33:42.641463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:42.641038    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 19:33:42.920099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:42.919999    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:33:43.642949 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:33:43.642908    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:33:43.646372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:43.646351    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:33:43.647432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:43.647410    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:33:43.959241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:43.959142    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb" podUID="cfd52fec-93ab-4d12-a8d7-38fdae88fa92" containerName="main" containerID="cri-o://19397e3b411fc6f628ff11789df9df0f334fd268e51b51ab360cea145a3638f4" gracePeriod=45
Oct 30 19:33:44.167320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.167278    6777 generic.go:356] "Generic (PLEG): container finished" podID="cfd52fec-93ab-4d12-a8d7-38fdae88fa92" containerID="19397e3b411fc6f628ff11789df9df0f334fd268e51b51ab360cea145a3638f4" exitCode=0
Oct 30 19:33:44.167499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.167315    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb" event={"ID":"cfd52fec-93ab-4d12-a8d7-38fdae88fa92","Type":"ContainerDied","Data":"19397e3b411fc6f628ff11789df9df0f334fd268e51b51ab360cea145a3638f4"}
Oct 30 19:33:44.319499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.319470    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:33:44.471277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.471236    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-dshm\") pod \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") "
Oct 30 19:33:44.471477 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.471347    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-kube-api-access-p9xv8\" (UniqueName: \"kubernetes.io/projected/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-kube-api-access-p9xv8\") pod \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") "
Oct 30 19:33:44.471477 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.471448    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-home\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-home\") pod \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") "
Oct 30 19:33:44.471673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.471653    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-model-cache\") pod \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") "
Oct 30 19:33:44.471737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.471705    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-home" pod "cfd52fec-93ab-4d12-a8d7-38fdae88fa92" (UID: "cfd52fec-93ab-4d12-a8d7-38fdae88fa92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:44.471853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.471831    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tmp-dir\") pod \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") "
Oct 30 19:33:44.472085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.472025    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tls-certs\") pod \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\" (UID: \"cfd52fec-93ab-4d12-a8d7-38fdae88fa92\") "
Oct 30 19:33:44.472085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.471890    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-model-cache" pod "cfd52fec-93ab-4d12-a8d7-38fdae88fa92" (UID: "cfd52fec-93ab-4d12-a8d7-38fdae88fa92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:44.472268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.472136    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tmp-dir" pod "cfd52fec-93ab-4d12-a8d7-38fdae88fa92" (UID: "cfd52fec-93ab-4d12-a8d7-38fdae88fa92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:44.472840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.472817    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:44.472840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.472840    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:44.473003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.472848    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:44.473550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.473526    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-dshm" pod "cfd52fec-93ab-4d12-a8d7-38fdae88fa92" (UID: "cfd52fec-93ab-4d12-a8d7-38fdae88fa92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:44.474037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.474015    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-kube-api-access-p9xv8" pod "cfd52fec-93ab-4d12-a8d7-38fdae88fa92" (UID: "cfd52fec-93ab-4d12-a8d7-38fdae88fa92"). InnerVolumeSpecName "kube-api-access-p9xv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:44.474175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.474139    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tls-certs" pod "cfd52fec-93ab-4d12-a8d7-38fdae88fa92" (UID: "cfd52fec-93ab-4d12-a8d7-38fdae88fa92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:44.574336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.574296    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:44.574336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.574331    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:44.574336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.574342    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9xv8\" (UniqueName: \"kubernetes.io/projected/cfd52fec-93ab-4d12-a8d7-38fdae88fa92-kube-api-access-p9xv8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:44.763037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.762992    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"]
Oct 30 19:33:44.764223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.764200    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c6d096e9-4314-49e3-a6bb-faa6a49e5563" containerName="revoke-keys"
Oct 30 19:33:44.764223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.764223    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d096e9-4314-49e3-a6bb-faa6a49e5563" containerName="revoke-keys"
Oct 30 19:33:44.764421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.764247    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cfd52fec-93ab-4d12-a8d7-38fdae88fa92" containerName="main"
Oct 30 19:33:44.764421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.764253    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd52fec-93ab-4d12-a8d7-38fdae88fa92" containerName="main"
Oct 30 19:33:44.764524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.764433    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd52fec-93ab-4d12-a8d7-38fdae88fa92" containerName="main"
Oct 30 19:33:44.764524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.764457    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d096e9-4314-49e3-a6bb-faa6a49e5563" containerName="revoke-keys"
Oct 30 19:33:44.768144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.768119    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:44.779337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.776394    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:33:44.779337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.776449    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:33:44.781246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.781216    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:44.781903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.781407    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jnqv\" (UniqueName: \"kubernetes.io/projected/dce30489-f500-4c3e-8931-6fa4e13c028a-kube-api-access-2jnqv\") pod \"maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:44.781903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.781455    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"]
Oct 30 19:33:44.781903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.781494    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:44.883210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.883110    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:44.883388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.883309    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:44.883471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.883427    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnqv\" (UniqueName: \"kubernetes.io/projected/dce30489-f500-4c3e-8931-6fa4e13c028a-kube-api-access-2jnqv\") pod \"maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:44.884164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.884046    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:44.885831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.885812    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:44.898029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:44.897993    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnqv\" (UniqueName: \"kubernetes.io/projected/dce30489-f500-4c3e-8931-6fa4e13c028a-kube-api-access-2jnqv\") pod \"maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:45.102195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:45.102154    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:45.182621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:45.182591    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb" event={"ID":"cfd52fec-93ab-4d12-a8d7-38fdae88fa92","Type":"ContainerDied","Data":"c6734b31db78dde4cf4cff0c945c3400d4488f7ef070e5136fd5db15a37df6b5"}
Oct 30 19:33:45.182763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:45.182648    6777 scope.go:122] "RemoveContainer" containerID="19397e3b411fc6f628ff11789df9df0f334fd268e51b51ab360cea145a3638f4"
Oct 30 19:33:45.182936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:45.182917    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"
Oct 30 19:33:45.219966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:45.219936    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"]
Oct 30 19:33:45.224731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:45.224700    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-891d80/test-model-891d80-kserve-8677659fff-49fzb"]
Oct 30 19:33:45.395580 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:33:45.395545    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce30489_f500_4c3e_8931_6fa4e13c028a.slice/crio-41aad0b253c6cf9d43034a808ef7447c24ec3df403fd90e938aa74ff89b7dcdf WatchSource:0}: Error finding container 41aad0b253c6cf9d43034a808ef7447c24ec3df403fd90e938aa74ff89b7dcdf: Status 404 returned error can't find the container with id 41aad0b253c6cf9d43034a808ef7447c24ec3df403fd90e938aa74ff89b7dcdf
Oct 30 19:33:45.396265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:45.396240    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"]
Oct 30 19:33:46.117190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:46.117147    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd52fec-93ab-4d12-a8d7-38fdae88fa92" path="/var/lib/kubelet/pods/cfd52fec-93ab-4d12-a8d7-38fdae88fa92/volumes"
Oct 30 19:33:46.192765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:46.192718    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch" event={"ID":"dce30489-f500-4c3e-8931-6fa4e13c028a","Type":"ContainerStarted","Data":"0d650fe88e87a8d358e628a106b12119a7ed1d1f38d772a1f9075d36457d543d"}
Oct 30 19:33:46.192958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:46.192781    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch" event={"ID":"dce30489-f500-4c3e-8931-6fa4e13c028a","Type":"ContainerStarted","Data":"41aad0b253c6cf9d43034a808ef7447c24ec3df403fd90e938aa74ff89b7dcdf"}
Oct 30 19:33:46.213623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:46.213545    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch" podStartSLOduration=2.213529534 podStartE2EDuration="2.213529534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:46.211781218 +0000 UTC m=+1608.208696827" watchObservedRunningTime="2026-10-30 19:33:46.213529534 +0000 UTC m=+1608.210444919"
Oct 30 19:33:47.204982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:47.204942    6777 generic.go:356] "Generic (PLEG): container finished" podID="dce30489-f500-4c3e-8931-6fa4e13c028a" containerID="0d650fe88e87a8d358e628a106b12119a7ed1d1f38d772a1f9075d36457d543d" exitCode=0
Oct 30 19:33:47.205547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:47.204980    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch" event={"ID":"dce30489-f500-4c3e-8931-6fa4e13c028a","Type":"ContainerDied","Data":"0d650fe88e87a8d358e628a106b12119a7ed1d1f38d772a1f9075d36457d543d"}
Oct 30 19:33:48.121953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.121916    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"]
Oct 30 19:33:48.583459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.583430    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:48.685999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.685902    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce30489-f500-4c3e-8931-6fa4e13c028a-kube-api-access-2jnqv\" (UniqueName: \"kubernetes.io/projected/dce30489-f500-4c3e-8931-6fa4e13c028a-kube-api-access-2jnqv\") pod \"dce30489-f500-4c3e-8931-6fa4e13c028a\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") "
Oct 30 19:33:48.686215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.686166    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-xks-ca\") pod \"dce30489-f500-4c3e-8931-6fa4e13c028a\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") "
Oct 30 19:33:48.686374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.686357    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-openshift-ca\") pod \"dce30489-f500-4c3e-8931-6fa4e13c028a\" (UID: \"dce30489-f500-4c3e-8931-6fa4e13c028a\") "
Oct 30 19:33:48.686826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.686794    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-openshift-ca" pod "dce30489-f500-4c3e-8931-6fa4e13c028a" (UID: "dce30489-f500-4c3e-8931-6fa4e13c028a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:48.687295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.687270    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:48.688235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.688213    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-xks-ca" pod "dce30489-f500-4c3e-8931-6fa4e13c028a" (UID: "dce30489-f500-4c3e-8931-6fa4e13c028a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:48.688442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.688417    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce30489-f500-4c3e-8931-6fa4e13c028a-kube-api-access-2jnqv" pod "dce30489-f500-4c3e-8931-6fa4e13c028a" (UID: "dce30489-f500-4c3e-8931-6fa4e13c028a"). InnerVolumeSpecName "kube-api-access-2jnqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:48.789427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.789377    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jnqv\" (UniqueName: \"kubernetes.io/projected/dce30489-f500-4c3e-8931-6fa4e13c028a-kube-api-access-2jnqv\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:48.789427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:48.789416    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce30489-f500-4c3e-8931-6fa4e13c028a-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:49.324124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:49.323793    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"
Oct 30 19:33:49.324124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:49.323849    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch" event={"ID":"dce30489-f500-4c3e-8931-6fa4e13c028a","Type":"ContainerDied","Data":"41aad0b253c6cf9d43034a808ef7447c24ec3df403fd90e938aa74ff89b7dcdf"}
Oct 30 19:33:49.324124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:49.323961    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41aad0b253c6cf9d43034a808ef7447c24ec3df403fd90e938aa74ff89b7dcdf"
Oct 30 19:33:50.645948 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:50.645890    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:33:53.292086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:53.286170    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.533645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:53.532872    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.556221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:53.533856    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx" podUID="f07f718a-19bb-4e4c-a4c4-f6a10b687225" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded"
Oct 30 19:33:53.841019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:53.840915    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:54.893626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:54.893579    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"]
Oct 30 19:33:54.924978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:54.924943    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"]
Oct 30 19:33:59.895201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:59.895042    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b" podUID="eb1df337-9ecf-43f5-95ad-99faa700a4b0" containerName="payload-processing" containerID="cri-o://40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda" gracePeriod=25
Oct 30 19:33:59.926146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:33:59.926020    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2" podUID="1db3bbf8-847c-4004-8b1b-91bca311dbf7" containerName="payload-pre-processing" containerID="cri-o://f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995" gracePeriod=25
Oct 30 19:34:02.089694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:02.089644    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"]
Oct 30 19:34:02.096594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:02.096555    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-891d80-6q6ch"]
Oct 30 19:34:02.118773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:02.118726    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce30489-f500-4c3e-8931-6fa4e13c028a" path="/var/lib/kubelet/pods/dce30489-f500-4c3e-8931-6fa4e13c028a/volumes"
Oct 30 19:34:02.952594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:02.952552    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2" podUID="1db3bbf8-847c-4004-8b1b-91bca311dbf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:02.952777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:02.952552    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b" podUID="eb1df337-9ecf-43f5-95ad-99faa700a4b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:03.224260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.224167    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf" podUID="76810fef-bb01-431d-b4a9-faf73e8f25ce" containerName="main" containerID="cri-o://b99b72adbf06c7ed0771d891c815317512c53fbd18fe706bbe953b0bf8278427" gracePeriod=45
Oct 30 19:34:03.339343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.339305    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"]
Oct 30 19:34:03.340523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.340491    6777 cpu_manager.go:414] "containerMap: removing container" podUID="dce30489-f500-4c3e-8931-6fa4e13c028a" containerName="revoke-keys"
Oct 30 19:34:03.340523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.340515    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce30489-f500-4c3e-8931-6fa4e13c028a" containerName="revoke-keys"
Oct 30 19:34:03.340718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.340681    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce30489-f500-4c3e-8931-6fa4e13c028a" containerName="revoke-keys"
Oct 30 19:34:03.417179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.416346    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"]
Oct 30 19:34:03.417179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.416478    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.425642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.425547    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:03.426028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.425982    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:03.448727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.447968    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggs82\" (UniqueName: \"kubernetes.io/projected/a71ca435-4994-4501-bb58-b1010e0c185e-kube-api-access-ggs82\") pod \"maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.448727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.448345    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.448727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.448405    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.550860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.550819    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.551077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.550873    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.551167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.551098    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggs82\" (UniqueName: \"kubernetes.io/projected/a71ca435-4994-4501-bb58-b1010e0c185e-kube-api-access-ggs82\") pod \"maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.551579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.551557    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.570480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.569535    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.570480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.569542    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggs82\" (UniqueName: \"kubernetes.io/projected/a71ca435-4994-4501-bb58-b1010e0c185e-kube-api-access-ggs82\") pod \"maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.748304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.748265    6777 generic.go:356] "Generic (PLEG): container finished" podID="76810fef-bb01-431d-b4a9-faf73e8f25ce" containerID="b99b72adbf06c7ed0771d891c815317512c53fbd18fe706bbe953b0bf8278427" exitCode=0
Oct 30 19:34:03.748463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.748339    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf" event={"ID":"76810fef-bb01-431d-b4a9-faf73e8f25ce","Type":"ContainerDied","Data":"b99b72adbf06c7ed0771d891c815317512c53fbd18fe706bbe953b0bf8278427"}
Oct 30 19:34:03.806470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.806383    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:03.806685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:03.806526    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf" podUID="76810fef-bb01-431d-b4a9-faf73e8f25ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.145:8000/ready\": dial tcp 10.134.0.145:8000: connect: connection refused"
Oct 30 19:34:04.260236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.260199    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:34:04.367450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.367359    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-home\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-home\") pod \"76810fef-bb01-431d-b4a9-faf73e8f25ce\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") "
Oct 30 19:34:04.367643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.367499    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-model-cache\") pod \"76810fef-bb01-431d-b4a9-faf73e8f25ce\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") "
Oct 30 19:34:04.367643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.367521    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76810fef-bb01-431d-b4a9-faf73e8f25ce-kube-api-access-khdb4\" (UniqueName: \"kubernetes.io/projected/76810fef-bb01-431d-b4a9-faf73e8f25ce-kube-api-access-khdb4\") pod \"76810fef-bb01-431d-b4a9-faf73e8f25ce\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") "
Oct 30 19:34:04.367643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.367575    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76810fef-bb01-431d-b4a9-faf73e8f25ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/76810fef-bb01-431d-b4a9-faf73e8f25ce-tls-certs\") pod \"76810fef-bb01-431d-b4a9-faf73e8f25ce\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") "
Oct 30 19:34:04.367822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.367689    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-home" pod "76810fef-bb01-431d-b4a9-faf73e8f25ce" (UID: "76810fef-bb01-431d-b4a9-faf73e8f25ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:04.367822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.367715    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-tmp-dir\") pod \"76810fef-bb01-431d-b4a9-faf73e8f25ce\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") "
Oct 30 19:34:04.367822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.367746    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-dshm\") pod \"76810fef-bb01-431d-b4a9-faf73e8f25ce\" (UID: \"76810fef-bb01-431d-b4a9-faf73e8f25ce\") "
Oct 30 19:34:04.367822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.367783    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-model-cache" pod "76810fef-bb01-431d-b4a9-faf73e8f25ce" (UID: "76810fef-bb01-431d-b4a9-faf73e8f25ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:04.368053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.367965    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-tmp-dir" pod "76810fef-bb01-431d-b4a9-faf73e8f25ce" (UID: "76810fef-bb01-431d-b4a9-faf73e8f25ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:04.368799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.368770    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:04.368799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.368803    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:04.368974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.368818    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:04.370147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.370124    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-dshm" pod "76810fef-bb01-431d-b4a9-faf73e8f25ce" (UID: "76810fef-bb01-431d-b4a9-faf73e8f25ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:04.370238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.370143    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76810fef-bb01-431d-b4a9-faf73e8f25ce-tls-certs" pod "76810fef-bb01-431d-b4a9-faf73e8f25ce" (UID: "76810fef-bb01-431d-b4a9-faf73e8f25ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:04.370238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.370182    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76810fef-bb01-431d-b4a9-faf73e8f25ce-kube-api-access-khdb4" pod "76810fef-bb01-431d-b4a9-faf73e8f25ce" (UID: "76810fef-bb01-431d-b4a9-faf73e8f25ce"). InnerVolumeSpecName "kube-api-access-khdb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:04.438228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.438194    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"]
Oct 30 19:34:04.438433 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:34:04.438404    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71ca435_4994_4501_bb58_b1010e0c185e.slice/crio-651328bcbbeffb7d005a6a06b5b242bdfa256be322e6e90d211dca227e42a0d0 WatchSource:0}: Error finding container 651328bcbbeffb7d005a6a06b5b242bdfa256be322e6e90d211dca227e42a0d0: Status 404 returned error can't find the container with id 651328bcbbeffb7d005a6a06b5b242bdfa256be322e6e90d211dca227e42a0d0
Oct 30 19:34:04.470589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.470558    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76810fef-bb01-431d-b4a9-faf73e8f25ce-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:04.470589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.470588    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khdb4\" (UniqueName: \"kubernetes.io/projected/76810fef-bb01-431d-b4a9-faf73e8f25ce-kube-api-access-khdb4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:04.470789 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.470598    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76810fef-bb01-431d-b4a9-faf73e8f25ce-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:04.758499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.758467    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"
Oct 30 19:34:04.758709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.758505    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf" event={"ID":"76810fef-bb01-431d-b4a9-faf73e8f25ce","Type":"ContainerDied","Data":"11dc8d2785d6060a67497f036c5ff08b36f8d597e15d9130c96117023a3f5fc0"}
Oct 30 19:34:04.758709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.758564    6777 scope.go:122] "RemoveContainer" containerID="b99b72adbf06c7ed0771d891c815317512c53fbd18fe706bbe953b0bf8278427"
Oct 30 19:34:04.760254 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.760223    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5" event={"ID":"a71ca435-4994-4501-bb58-b1010e0c185e","Type":"ContainerStarted","Data":"651328bcbbeffb7d005a6a06b5b242bdfa256be322e6e90d211dca227e42a0d0"}
Oct 30 19:34:04.789771 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.789736    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"]
Oct 30 19:34:04.797200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:04.797143    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-33b4d7/test-model-33b4d7-kserve-6d995b4df-4qmhf"]
Oct 30 19:34:05.772586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:05.772541    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5" event={"ID":"a71ca435-4994-4501-bb58-b1010e0c185e","Type":"ContainerStarted","Data":"40cf458f41665c377b90c9dbdd7496cb9d894d85fe2075d826d7da8b705ff02b"}
Oct 30 19:34:05.793295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:05.793234    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5" podStartSLOduration=2.793216729 podStartE2EDuration="2.793216729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:05.790457046 +0000 UTC m=+1627.787372432" watchObservedRunningTime="2026-10-30 19:34:05.793216729 +0000 UTC m=+1627.790132114"
Oct 30 19:34:06.116994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:06.116959    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76810fef-bb01-431d-b4a9-faf73e8f25ce" path="/var/lib/kubelet/pods/76810fef-bb01-431d-b4a9-faf73e8f25ce/volumes"
Oct 30 19:34:06.787157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:06.787122    6777 generic.go:356] "Generic (PLEG): container finished" podID="a71ca435-4994-4501-bb58-b1010e0c185e" containerID="40cf458f41665c377b90c9dbdd7496cb9d894d85fe2075d826d7da8b705ff02b" exitCode=0
Oct 30 19:34:06.787674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:06.787187    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5" event={"ID":"a71ca435-4994-4501-bb58-b1010e0c185e","Type":"ContainerDied","Data":"40cf458f41665c377b90c9dbdd7496cb9d894d85fe2075d826d7da8b705ff02b"}
Oct 30 19:34:07.640264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:07.640224    6777 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:34:07.640484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:07.640311    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:08.487328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.487294    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:08.612848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.612759    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71ca435-4994-4501-bb58-b1010e0c185e-kube-api-access-ggs82\" (UniqueName: \"kubernetes.io/projected/a71ca435-4994-4501-bb58-b1010e0c185e-kube-api-access-ggs82\") pod \"a71ca435-4994-4501-bb58-b1010e0c185e\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") "
Oct 30 19:34:08.612848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.612817    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-openshift-ca\") pod \"a71ca435-4994-4501-bb58-b1010e0c185e\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") "
Oct 30 19:34:08.613148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.612871    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-xks-ca\") pod \"a71ca435-4994-4501-bb58-b1010e0c185e\" (UID: \"a71ca435-4994-4501-bb58-b1010e0c185e\") "
Oct 30 19:34:08.613292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.613265    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-openshift-ca" pod "a71ca435-4994-4501-bb58-b1010e0c185e" (UID: "a71ca435-4994-4501-bb58-b1010e0c185e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:08.613858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.613838    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:08.615145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.615113    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71ca435-4994-4501-bb58-b1010e0c185e-kube-api-access-ggs82" pod "a71ca435-4994-4501-bb58-b1010e0c185e" (UID: "a71ca435-4994-4501-bb58-b1010e0c185e"). InnerVolumeSpecName "kube-api-access-ggs82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:08.615394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.615372    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-xks-ca" pod "a71ca435-4994-4501-bb58-b1010e0c185e" (UID: "a71ca435-4994-4501-bb58-b1010e0c185e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:08.715739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.715704    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggs82\" (UniqueName: \"kubernetes.io/projected/a71ca435-4994-4501-bb58-b1010e0c185e-kube-api-access-ggs82\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:08.715739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.715735    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ca435-4994-4501-bb58-b1010e0c185e-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:08.821641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.821604    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"
Oct 30 19:34:08.822137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.822107    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5" event={"ID":"a71ca435-4994-4501-bb58-b1010e0c185e","Type":"ContainerDied","Data":"651328bcbbeffb7d005a6a06b5b242bdfa256be322e6e90d211dca227e42a0d0"}
Oct 30 19:34:08.822227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:08.822183    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="651328bcbbeffb7d005a6a06b5b242bdfa256be322e6e90d211dca227e42a0d0"
Oct 30 19:34:10.639008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:10.638951    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"]
Oct 30 19:34:10.640027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:10.639407    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy" containerID="cri-o://c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8" gracePeriod=30
Oct 30 19:34:10.929083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:10.928667    6777 generic.go:356] "Generic (PLEG): container finished" podID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerID="1757d769dc961492da3aaeaa70127548e1e1e506bc4ddd83dd248895377fcdf0" exitCode=0
Oct 30 19:34:10.929083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:10.928749    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" event={"ID":"cc320727-adce-4d5f-bab8-3b6806c955e0","Type":"ContainerDied","Data":"1757d769dc961492da3aaeaa70127548e1e1e506bc4ddd83dd248895377fcdf0"}
Oct 30 19:34:10.933776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:10.929512    6777 scope.go:122] "RemoveContainer" containerID="1757d769dc961492da3aaeaa70127548e1e1e506bc4ddd83dd248895377fcdf0"
Oct 30 19:34:10.933776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:10.932775    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:10.933776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:10.932869    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:10.933776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:10.932921    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:11.107508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.107474    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"]
Oct 30 19:34:11.108444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.108426    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a71ca435-4994-4501-bb58-b1010e0c185e" containerName="revoke-keys"
Oct 30 19:34:11.108531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.108447    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71ca435-4994-4501-bb58-b1010e0c185e" containerName="revoke-keys"
Oct 30 19:34:11.108531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.108467    6777 cpu_manager.go:414] "containerMap: removing container" podUID="76810fef-bb01-431d-b4a9-faf73e8f25ce" containerName="main"
Oct 30 19:34:11.108531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.108474    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76810fef-bb01-431d-b4a9-faf73e8f25ce" containerName="main"
Oct 30 19:34:11.108722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.108704    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71ca435-4994-4501-bb58-b1010e0c185e" containerName="revoke-keys"
Oct 30 19:34:11.108784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.108727    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="76810fef-bb01-431d-b4a9-faf73e8f25ce" containerName="main"
Oct 30 19:34:11.112628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.112605    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.131800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.131767    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"]
Oct 30 19:34:11.133827 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.133796    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:11.133987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.133855    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:11.312815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.312771    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.313007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.312960    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgmdf\" (UniqueName: \"kubernetes.io/projected/9617810d-7469-4ddf-8f8c-0756ff830f2e-kube-api-access-zgmdf\") pod \"maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.313247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.313222    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.414944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.414906    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.415177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.415007    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgmdf\" (UniqueName: \"kubernetes.io/projected/9617810d-7469-4ddf-8f8c-0756ff830f2e-kube-api-access-zgmdf\") pod \"maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.415177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.415136    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.415630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.415603    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.417528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.417504    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.436352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.436322    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgmdf\" (UniqueName: \"kubernetes.io/projected/9617810d-7469-4ddf-8f8c-0756ff830f2e-kube-api-access-zgmdf\") pod \"maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.741486 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.741395    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:11.945733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.945691    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" event={"ID":"cc320727-adce-4d5f-bab8-3b6806c955e0","Type":"ContainerStarted","Data":"ae5c32ac92ee615b8d39509366f51e304d06eb2981e152c1240c53b2440757cb"}
Oct 30 19:34:11.946285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:11.946267    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:34:12.125585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:12.125555    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"]
Oct 30 19:34:12.125839 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:34:12.125810    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9617810d_7469_4ddf_8f8c_0756ff830f2e.slice/crio-a200d4d94660687ca788952f326fca664371e8270200728b0472fb3d9a7e8ef4 WatchSource:0}: Error finding container a200d4d94660687ca788952f326fca664371e8270200728b0472fb3d9a7e8ef4: Status 404 returned error can't find the container with id a200d4d94660687ca788952f326fca664371e8270200728b0472fb3d9a7e8ef4
Oct 30 19:34:12.704630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:12.704590    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2" podUID="1db3bbf8-847c-4004-8b1b-91bca311dbf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:12.704820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:12.704638    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b" podUID="eb1df337-9ecf-43f5-95ad-99faa700a4b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:12.946580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:12.946542    6777 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:34:12.947127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:12.946605    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 19:34:12.981559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:12.981469    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj" event={"ID":"9617810d-7469-4ddf-8f8c-0756ff830f2e","Type":"ContainerStarted","Data":"df804c565fd63b9045b6190b3c4ed04bcf6090516c5c11886793da2cfa4cd4e3"}
Oct 30 19:34:12.981559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:12.981533    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj" event={"ID":"9617810d-7469-4ddf-8f8c-0756ff830f2e","Type":"ContainerStarted","Data":"a200d4d94660687ca788952f326fca664371e8270200728b0472fb3d9a7e8ef4"}
Oct 30 19:34:12.991011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:12.990968    6777 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:34:12.991167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:12.991032    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:34:13.001930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:13.001873    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj" podStartSLOduration=2.001854499 podStartE2EDuration="2.001854499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:12.99966858 +0000 UTC m=+1634.996583986" watchObservedRunningTime="2026-10-30 19:34:13.001854499 +0000 UTC m=+1634.998769884"
Oct 30 19:34:13.539137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:13.539100    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"]
Oct 30 19:34:13.572084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:13.567857    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"]
Oct 30 19:34:13.728708 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:34:13.728611    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:34:13.950331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:13.947649    6777 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:34:13.950331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:13.947716    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 19:34:14.115082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:14.114036    6777 generic.go:356] "Generic (PLEG): container finished" podID="9617810d-7469-4ddf-8f8c-0756ff830f2e" containerID="df804c565fd63b9045b6190b3c4ed04bcf6090516c5c11886793da2cfa4cd4e3" exitCode=0
Oct 30 19:34:14.213737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:14.213691    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj" event={"ID":"9617810d-7469-4ddf-8f8c-0756ff830f2e","Type":"ContainerDied","Data":"df804c565fd63b9045b6190b3c4ed04bcf6090516c5c11886793da2cfa4cd4e3"}
Oct 30 19:34:14.947268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:14.947229    6777 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:34:14.947487 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:14.947293    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 19:34:15.497704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.497666    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:15.529616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.529577    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-openshift-ca\") pod \"9617810d-7469-4ddf-8f8c-0756ff830f2e\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") "
Oct 30 19:34:15.529818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.529778    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-xks-ca\") pod \"9617810d-7469-4ddf-8f8c-0756ff830f2e\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") "
Oct 30 19:34:15.529904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.529856    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9617810d-7469-4ddf-8f8c-0756ff830f2e-kube-api-access-zgmdf\" (UniqueName: \"kubernetes.io/projected/9617810d-7469-4ddf-8f8c-0756ff830f2e-kube-api-access-zgmdf\") pod \"9617810d-7469-4ddf-8f8c-0756ff830f2e\" (UID: \"9617810d-7469-4ddf-8f8c-0756ff830f2e\") "
Oct 30 19:34:15.529994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.529969    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-openshift-ca" pod "9617810d-7469-4ddf-8f8c-0756ff830f2e" (UID: "9617810d-7469-4ddf-8f8c-0756ff830f2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:15.530793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.530768    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:15.532151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.532114    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-xks-ca" pod "9617810d-7469-4ddf-8f8c-0756ff830f2e" (UID: "9617810d-7469-4ddf-8f8c-0756ff830f2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:15.532505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.532469    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9617810d-7469-4ddf-8f8c-0756ff830f2e-kube-api-access-zgmdf" pod "9617810d-7469-4ddf-8f8c-0756ff830f2e" (UID: "9617810d-7469-4ddf-8f8c-0756ff830f2e"). InnerVolumeSpecName "kube-api-access-zgmdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:15.632186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.632151    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9617810d-7469-4ddf-8f8c-0756ff830f2e-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:15.632186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.632182    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgmdf\" (UniqueName: \"kubernetes.io/projected/9617810d-7469-4ddf-8f8c-0756ff830f2e-kube-api-access-zgmdf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:15.949086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.947908    6777 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:34:15.949086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.947969    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 19:34:15.954931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:15.953401    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:34:16.029196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.029155    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"]
Oct 30 19:34:16.030558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.030518    6777 cpu_manager.go:414] "containerMap: removing container" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy"
Oct 30 19:34:16.030558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.030547    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy"
Oct 30 19:34:16.030715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.030634    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9617810d-7469-4ddf-8f8c-0756ff830f2e" containerName="revoke-keys"
Oct 30 19:34:16.030715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.030643    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9617810d-7469-4ddf-8f8c-0756ff830f2e" containerName="revoke-keys"
Oct 30 19:34:16.034893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.030913    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9617810d-7469-4ddf-8f8c-0756ff830f2e" containerName="revoke-keys"
Oct 30 19:34:16.034893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.030944    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="07890996-9067-4a7d-934c-333cd93d4400" containerName="istio-proxy"
Oct 30 19:34:16.038962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.038934    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"]
Oct 30 19:34:16.039101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.039034    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.061380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.057223    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-credential-socket\") pod \"07890996-9067-4a7d-934c-333cd93d4400\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") "
Oct 30 19:34:16.061380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.057350    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-socket\") pod \"07890996-9067-4a7d-934c-333cd93d4400\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") "
Oct 30 19:34:16.061380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.057397    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-kube-api-access-dc8kw\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-kube-api-access-dc8kw\") pod \"07890996-9067-4a7d-934c-333cd93d4400\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") "
Oct 30 19:34:16.061380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.057446    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-istio-token\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-istio-token\") pod \"07890996-9067-4a7d-934c-333cd93d4400\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") "
Oct 30 19:34:16.061380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.057645    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07890996-9067-4a7d-934c-333cd93d4400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07890996-9067-4a7d-934c-333cd93d4400-istiod-ca-cert\") pod \"07890996-9067-4a7d-934c-333cd93d4400\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") "
Oct 30 19:34:16.061380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.057769    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-certs\") pod \"07890996-9067-4a7d-934c-333cd93d4400\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") "
Oct 30 19:34:16.061380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.057819    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-envoy\") pod \"07890996-9067-4a7d-934c-333cd93d4400\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") "
Oct 30 19:34:16.061380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.057913    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07890996-9067-4a7d-934c-333cd93d4400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07890996-9067-4a7d-934c-333cd93d4400-istio-podinfo\") pod \"07890996-9067-4a7d-934c-333cd93d4400\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") "
Oct 30 19:34:16.061380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.058004    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-data\") pod \"07890996-9067-4a7d-934c-333cd93d4400\" (UID: \"07890996-9067-4a7d-934c-333cd93d4400\") "
Oct 30 19:34:16.079711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.079662    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-data" pod "07890996-9067-4a7d-934c-333cd93d4400" (UID: "07890996-9067-4a7d-934c-333cd93d4400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.079923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.079901    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-credential-socket" pod "07890996-9067-4a7d-934c-333cd93d4400" (UID: "07890996-9067-4a7d-934c-333cd93d4400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.080180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.080155    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-socket" pod "07890996-9067-4a7d-934c-333cd93d4400" (UID: "07890996-9067-4a7d-934c-333cd93d4400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.088740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.088697    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-kube-api-access-dc8kw" pod "07890996-9067-4a7d-934c-333cd93d4400" (UID: "07890996-9067-4a7d-934c-333cd93d4400"). InnerVolumeSpecName "kube-api-access-dc8kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:16.089363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.089332    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-certs" pod "07890996-9067-4a7d-934c-333cd93d4400" (UID: "07890996-9067-4a7d-934c-333cd93d4400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.089518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.089497    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07890996-9067-4a7d-934c-333cd93d4400-istiod-ca-cert" pod "07890996-9067-4a7d-934c-333cd93d4400" (UID: "07890996-9067-4a7d-934c-333cd93d4400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:16.094547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.094513    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07890996-9067-4a7d-934c-333cd93d4400-istio-podinfo" pod "07890996-9067-4a7d-934c-333cd93d4400" (UID: "07890996-9067-4a7d-934c-333cd93d4400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:16.094547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.094515    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-istio-token" pod "07890996-9067-4a7d-934c-333cd93d4400" (UID: "07890996-9067-4a7d-934c-333cd93d4400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:16.095865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.095832    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-envoy" pod "07890996-9067-4a7d-934c-333cd93d4400" (UID: "07890996-9067-4a7d-934c-333cd93d4400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.186393    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.186499    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-credential-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.186613    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istiod-ca-cert\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.186714    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-token\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.186901    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-data\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187096    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwv8\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-kube-api-access-tqwv8\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187165    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-podinfo\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187212    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-envoy\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187238    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-certs\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187451    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187469    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc8kw\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-kube-api-access-dc8kw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187484    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07890996-9067-4a7d-934c-333cd93d4400-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187499    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07890996-9067-4a7d-934c-333cd93d4400-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187513    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187527    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187540    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07890996-9067-4a7d-934c-333cd93d4400-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187554    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.188504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.187568    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07890996-9067-4a7d-934c-333cd93d4400-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.199883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.199372    6777 generic.go:356] "Generic (PLEG): container finished" podID="07890996-9067-4a7d-934c-333cd93d4400" containerID="c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8" exitCode=0
Oct 30 19:34:16.199883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.199440    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"
Oct 30 19:34:16.199883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.199454    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" event={"ID":"07890996-9067-4a7d-934c-333cd93d4400","Type":"ContainerDied","Data":"c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8"}
Oct 30 19:34:16.199883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.199517    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6" event={"ID":"07890996-9067-4a7d-934c-333cd93d4400","Type":"ContainerDied","Data":"1b2918b4685abedbc21d791756abac90557dd3ed238ec3b125e961eaf6e8c4ff"}
Oct 30 19:34:16.199883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.199562    6777 scope.go:122] "RemoveContainer" containerID="c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8"
Oct 30 19:34:16.206271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.205101    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj" event={"ID":"9617810d-7469-4ddf-8f8c-0756ff830f2e","Type":"ContainerDied","Data":"a200d4d94660687ca788952f326fca664371e8270200728b0472fb3d9a7e8ef4"}
Oct 30 19:34:16.206271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.205169    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a200d4d94660687ca788952f326fca664371e8270200728b0472fb3d9a7e8ef4"
Oct 30 19:34:16.206271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.205268    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"
Oct 30 19:34:16.221488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.221459    6777 scope.go:122] "RemoveContainer" containerID="c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8"
Oct 30 19:34:16.222014 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:34:16.221976    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8\": container with ID starting with c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8 not found: ID does not exist" containerID="c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8"
Oct 30 19:34:16.222206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.222180    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8"} err="failed to get container status \"c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8\": rpc error: code = NotFound desc = could not find container \"c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8\": container with ID starting with c4317d5f327056788e560d451e490d65d15b6e58f7f1a88fb2e3d0106a5260e8 not found: ID does not exist"
Oct 30 19:34:16.240183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.240048    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"]
Oct 30 19:34:16.245636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.245601    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-76fz6"]
Oct 30 19:34:16.288729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.288690    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-credential-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.288905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.288786    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istiod-ca-cert\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.288905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.288823    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-token\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289011    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-data\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289146    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-credential-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289186    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwv8\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-kube-api-access-tqwv8\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289247    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-podinfo\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289410    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-data\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289554    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istiod-ca-cert\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289457    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-envoy\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289633    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-certs\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289770 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289756    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.289920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.289897    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-certs\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.290253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.290225    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.291761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.291740    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-envoy\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.292367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.292348    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-podinfo\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.303384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.303353    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-token\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.303563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.303446    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwv8\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-kube-api-access-tqwv8\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.399110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.398150    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:16.785922 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:34:16.785885    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec1d99b_4ec2_4e26_8eed_0aae46a24b24.slice/crio-f3896c4814216d6fc4ac7f5326d2ad784b5fe4f6d5feb5b41273e3247d45b916 WatchSource:0}: Error finding container f3896c4814216d6fc4ac7f5326d2ad784b5fe4f6d5feb5b41273e3247d45b916: Status 404 returned error can't find the container with id f3896c4814216d6fc4ac7f5326d2ad784b5fe4f6d5feb5b41273e3247d45b916
Oct 30 19:34:16.786738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.786709    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"]
Oct 30 19:34:16.790572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.790516    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:16.790703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.790631    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:16.790703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.790676    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:16.948118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.947102    6777 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:34:16.948118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:16.947177    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 19:34:17.380334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.380291    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" event={"ID":"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24","Type":"ContainerStarted","Data":"68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172"}
Oct 30 19:34:17.380598 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.380579    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" event={"ID":"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24","Type":"ContainerStarted","Data":"f3896c4814216d6fc4ac7f5326d2ad784b5fe4f6d5feb5b41273e3247d45b916"}
Oct 30 19:34:17.408967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.408825    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:17.422602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.422563    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:34:17.436363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.436321    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 19:34:17.436836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.435752    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"]
Oct 30 19:34:17.489086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.488138    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" podStartSLOduration=5.488113733 podStartE2EDuration="5.488113733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:17.435589711 +0000 UTC m=+1639.432505097" watchObservedRunningTime="2026-10-30 19:34:17.488113733 +0000 UTC m=+1639.485029122"
Oct 30 19:34:17.497184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.497139    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-33b4d7-trmn5"]
Oct 30 19:34:17.947790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.947752    6777 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:34:17.948323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:17.947841    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 19:34:18.126428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.126381    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07890996-9067-4a7d-934c-333cd93d4400" path="/var/lib/kubelet/pods/07890996-9067-4a7d-934c-333cd93d4400/volumes"
Oct 30 19:34:18.127375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.127345    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a71ca435-4994-4501-bb58-b1010e0c185e" path="/var/lib/kubelet/pods/a71ca435-4994-4501-bb58-b1010e0c185e/volumes"
Oct 30 19:34:18.398931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.398897    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:34:18.399259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.399231    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 19:34:18.445123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.445032    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"]
Oct 30 19:34:18.467081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.464750    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"]
Oct 30 19:34:18.467081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.464884    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.470179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.470148    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:18.471455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.471429    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:18.532902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.532847    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b6a7368-cmqbs\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.533129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.533036    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b6a7368-cmqbs\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.533244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.533208    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sg8p\" (UniqueName: \"kubernetes.io/projected/9239a6b1-236b-4e72-a8ae-fabc64d22619-kube-api-access-2sg8p\") pod \"maas-api-revoke-sub-4b6a7368-cmqbs\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.539506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.539455    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw" podUID="fef5a740-4ced-428e-bb19-9092a9ae471f" containerName="payload-processing" containerID="cri-o://f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2" gracePeriod=25
Oct 30 19:34:18.579277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.579210    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp" podUID="cc79b5d0-dbf8-413d-ad18-216d07d3ad08" containerName="payload-pre-processing" containerID="cri-o://e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5" gracePeriod=25
Oct 30 19:34:18.635134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.635094    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b6a7368-cmqbs\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.635358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.635253    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sg8p\" (UniqueName: \"kubernetes.io/projected/9239a6b1-236b-4e72-a8ae-fabc64d22619-kube-api-access-2sg8p\") pod \"maas-api-revoke-sub-4b6a7368-cmqbs\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.635358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.635352    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b6a7368-cmqbs\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.635863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.635834    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b6a7368-cmqbs\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.637963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.637930    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b6a7368-cmqbs\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.656193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.648507    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sg8p\" (UniqueName: \"kubernetes.io/projected/9239a6b1-236b-4e72-a8ae-fabc64d22619-kube-api-access-2sg8p\") pod \"maas-api-revoke-sub-4b6a7368-cmqbs\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.843414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.841798    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:18.956415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.956384    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:34:18.958117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.958078    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:34:18.989027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:18.988561    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:34:19.199274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:19.199242    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"]
Oct 30 19:34:19.200723 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:34:19.200696    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9239a6b1_236b_4e72_a8ae_fabc64d22619.slice/crio-7fc77628e8a077c54a8cea728f62a85284e9b0c01764ac8f4e7af4dfc0913a7c WatchSource:0}: Error finding container 7fc77628e8a077c54a8cea728f62a85284e9b0c01764ac8f4e7af4dfc0913a7c: Status 404 returned error can't find the container with id 7fc77628e8a077c54a8cea728f62a85284e9b0c01764ac8f4e7af4dfc0913a7c
Oct 30 19:34:19.409492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:19.409455    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:34:19.409758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:19.409731    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 19:34:19.447675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:19.447625    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs" event={"ID":"9239a6b1-236b-4e72-a8ae-fabc64d22619","Type":"ContainerStarted","Data":"7fc77628e8a077c54a8cea728f62a85284e9b0c01764ac8f4e7af4dfc0913a7c"}
Oct 30 19:34:20.382671 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.382642    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:34:20.398742 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.398705    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:34:20.398957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.398936    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 19:34:20.418396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.418364    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.495681    6777 generic.go:356] "Generic (PLEG): container finished" podID="eb1df337-9ecf-43f5-95ad-99faa700a4b0" containerID="40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda" exitCode=0
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.495735    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.496019    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b" event={"ID":"eb1df337-9ecf-43f5-95ad-99faa700a4b0","Type":"ContainerDied","Data":"40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda"}
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.496127    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b" event={"ID":"eb1df337-9ecf-43f5-95ad-99faa700a4b0","Type":"ContainerDied","Data":"336a80e40d25a4c27e2bb11b34f5e8517616ee506e80d3ca0a48903e737a02de"}
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.496164    6777 scope.go:122] "RemoveContainer" containerID="40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda"
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.496889    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db3bbf8-847c-4004-8b1b-91bca311dbf7-kube-api-access-wkk62\" (UniqueName: \"kubernetes.io/projected/1db3bbf8-847c-4004-8b1b-91bca311dbf7-kube-api-access-wkk62\") pod \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\" (UID: \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\") "
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.496977    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1df337-9ecf-43f5-95ad-99faa700a4b0-kube-api-access-h9ggg\" (UniqueName: \"kubernetes.io/projected/eb1df337-9ecf-43f5-95ad-99faa700a4b0-kube-api-access-h9ggg\") pod \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\" (UID: \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\") "
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.497036    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1df337-9ecf-43f5-95ad-99faa700a4b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1df337-9ecf-43f5-95ad-99faa700a4b0-plugins-config-volume\") pod \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\" (UID: \"eb1df337-9ecf-43f5-95ad-99faa700a4b0\") "
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.497218    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db3bbf8-847c-4004-8b1b-91bca311dbf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db3bbf8-847c-4004-8b1b-91bca311dbf7-plugins-config-volume\") pod \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\" (UID: \"1db3bbf8-847c-4004-8b1b-91bca311dbf7\") "
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.498588    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db3bbf8-847c-4004-8b1b-91bca311dbf7-plugins-config-volume" pod "1db3bbf8-847c-4004-8b1b-91bca311dbf7" (UID: "1db3bbf8-847c-4004-8b1b-91bca311dbf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.498839    6777 generic.go:356] "Generic (PLEG): container finished" podID="1db3bbf8-847c-4004-8b1b-91bca311dbf7" containerID="f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995" exitCode=0
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.498981    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2" event={"ID":"1db3bbf8-847c-4004-8b1b-91bca311dbf7","Type":"ContainerDied","Data":"f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995"}
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.499034    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2" event={"ID":"1db3bbf8-847c-4004-8b1b-91bca311dbf7","Type":"ContainerDied","Data":"a5d0ba2c48d35732573d6070be6f4cce7ea973e26d01d6ec098f0a7fb726fb96"}
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.499126    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.499414    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db3bbf8-847c-4004-8b1b-91bca311dbf7-kube-api-access-wkk62" pod "1db3bbf8-847c-4004-8b1b-91bca311dbf7" (UID: "1db3bbf8-847c-4004-8b1b-91bca311dbf7"). InnerVolumeSpecName "kube-api-access-wkk62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:20.499722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.499733    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1df337-9ecf-43f5-95ad-99faa700a4b0-plugins-config-volume" pod "eb1df337-9ecf-43f5-95ad-99faa700a4b0" (UID: "eb1df337-9ecf-43f5-95ad-99faa700a4b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:20.502933 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.501629    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs" event={"ID":"9239a6b1-236b-4e72-a8ae-fabc64d22619","Type":"ContainerStarted","Data":"3e4c9f4400fe1735ebceb2367360ec1fa51df8d022359d8da8ba10a9c601f80a"}
Oct 30 19:34:20.506190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.506159    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1df337-9ecf-43f5-95ad-99faa700a4b0-kube-api-access-h9ggg" pod "eb1df337-9ecf-43f5-95ad-99faa700a4b0" (UID: "eb1df337-9ecf-43f5-95ad-99faa700a4b0"). InnerVolumeSpecName "kube-api-access-h9ggg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:20.527287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.527231    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs" podStartSLOduration=2.527217516 podStartE2EDuration="2.527217516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:20.522243565 +0000 UTC m=+1642.519158952" watchObservedRunningTime="2026-10-30 19:34:20.527217516 +0000 UTC m=+1642.524132900"
Oct 30 19:34:20.541882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.541754    6777 scope.go:122] "RemoveContainer" containerID="40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda"
Oct 30 19:34:20.542205 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:34:20.542182    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda\": container with ID starting with 40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda not found: ID does not exist" containerID="40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda"
Oct 30 19:34:20.542357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.542327    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda"} err="failed to get container status \"40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda\": rpc error: code = NotFound desc = could not find container \"40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda\": container with ID starting with 40ba27739e27033d43f312212e41ba440e80d0556a842e663234a39653242bda not found: ID does not exist"
Oct 30 19:34:20.542357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.542356    6777 scope.go:122] "RemoveContainer" containerID="f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995"
Oct 30 19:34:20.582946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.582917    6777 scope.go:122] "RemoveContainer" containerID="f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995"
Oct 30 19:34:20.583377 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:34:20.583348    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995\": container with ID starting with f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995 not found: ID does not exist" containerID="f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995"
Oct 30 19:34:20.583445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.583384    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995"} err="failed to get container status \"f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995\": rpc error: code = NotFound desc = could not find container \"f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995\": container with ID starting with f93dca6db7c568051f1bba1929363b20cf1f44cabd61c274992adc0f678f9995 not found: ID does not exist"
Oct 30 19:34:20.600300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.600265    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkk62\" (UniqueName: \"kubernetes.io/projected/1db3bbf8-847c-4004-8b1b-91bca311dbf7-kube-api-access-wkk62\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:20.600300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.600298    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9ggg\" (UniqueName: \"kubernetes.io/projected/eb1df337-9ecf-43f5-95ad-99faa700a4b0-kube-api-access-h9ggg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:20.600300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.600311    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1df337-9ecf-43f5-95ad-99faa700a4b0-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:20.600563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.600344    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db3bbf8-847c-4004-8b1b-91bca311dbf7-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:20.823140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.823098    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"]
Oct 30 19:34:20.829176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.829144    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-891d80-78cbdfb8c6-9488b"]
Oct 30 19:34:20.843760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.843726    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"]
Oct 30 19:34:20.856090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:20.848472    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-891d80-5959cbd4dc-4snm2"]
Oct 30 19:34:21.226713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:21.226616    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"]
Oct 30 19:34:21.399498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:21.399468    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:34:21.399887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:21.399524    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 19:34:21.606099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:21.602459    6777 generic.go:356] "Generic (PLEG): container finished" podID="9239a6b1-236b-4e72-a8ae-fabc64d22619" containerID="3e4c9f4400fe1735ebceb2367360ec1fa51df8d022359d8da8ba10a9c601f80a" exitCode=0
Oct 30 19:34:21.606099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:21.602762    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs" event={"ID":"9239a6b1-236b-4e72-a8ae-fabc64d22619","Type":"ContainerDied","Data":"3e4c9f4400fe1735ebceb2367360ec1fa51df8d022359d8da8ba10a9c601f80a"}
Oct 30 19:34:22.117995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:22.117960    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db3bbf8-847c-4004-8b1b-91bca311dbf7" path="/var/lib/kubelet/pods/1db3bbf8-847c-4004-8b1b-91bca311dbf7/volumes"
Oct 30 19:34:22.118540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:22.118525    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1df337-9ecf-43f5-95ad-99faa700a4b0" path="/var/lib/kubelet/pods/eb1df337-9ecf-43f5-95ad-99faa700a4b0/volumes"
Oct 30 19:34:22.399366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:22.399280    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:34:22.399366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:22.399339    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 19:34:23.042087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.038713    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:23.169485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.163038    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-openshift-ca\") pod \"9239a6b1-236b-4e72-a8ae-fabc64d22619\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") "
Oct 30 19:34:23.169485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.163208    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9239a6b1-236b-4e72-a8ae-fabc64d22619-kube-api-access-2sg8p\" (UniqueName: \"kubernetes.io/projected/9239a6b1-236b-4e72-a8ae-fabc64d22619-kube-api-access-2sg8p\") pod \"9239a6b1-236b-4e72-a8ae-fabc64d22619\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") "
Oct 30 19:34:23.169485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.163457    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-xks-ca\") pod \"9239a6b1-236b-4e72-a8ae-fabc64d22619\" (UID: \"9239a6b1-236b-4e72-a8ae-fabc64d22619\") "
Oct 30 19:34:23.169485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.165326    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-openshift-ca" pod "9239a6b1-236b-4e72-a8ae-fabc64d22619" (UID: "9239a6b1-236b-4e72-a8ae-fabc64d22619"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:23.211080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.210205    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-xks-ca" pod "9239a6b1-236b-4e72-a8ae-fabc64d22619" (UID: "9239a6b1-236b-4e72-a8ae-fabc64d22619"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:23.221089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.213770    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9239a6b1-236b-4e72-a8ae-fabc64d22619-kube-api-access-2sg8p" pod "9239a6b1-236b-4e72-a8ae-fabc64d22619" (UID: "9239a6b1-236b-4e72-a8ae-fabc64d22619"). InnerVolumeSpecName "kube-api-access-2sg8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:23.299284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.298572    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sg8p\" (UniqueName: \"kubernetes.io/projected/9239a6b1-236b-4e72-a8ae-fabc64d22619-kube-api-access-2sg8p\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:23.299284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.298610    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:23.299284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.298625    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9239a6b1-236b-4e72-a8ae-fabc64d22619-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:23.323995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.323873    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"]
Oct 30 19:34:23.329398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.329366    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef5dca-dvcfj"]
Oct 30 19:34:23.439301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.434447    6777 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:34:23.439301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.434514    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 19:34:23.634744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.634656    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"
Oct 30 19:34:23.634744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.634693    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs" event={"ID":"9239a6b1-236b-4e72-a8ae-fabc64d22619","Type":"ContainerDied","Data":"7fc77628e8a077c54a8cea728f62a85284e9b0c01764ac8f4e7af4dfc0913a7c"}
Oct 30 19:34:23.634981 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:23.634751    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fc77628e8a077c54a8cea728f62a85284e9b0c01764ac8f4e7af4dfc0913a7c"
Oct 30 19:34:24.115484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:24.115450    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9617810d-7469-4ddf-8f8c-0756ff830f2e" path="/var/lib/kubelet/pods/9617810d-7469-4ddf-8f8c-0756ff830f2e/volumes"
Oct 30 19:34:24.556843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:24.556808    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:24.557385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:24.557352    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:24.558399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:24.558378    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:34:26.603645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:26.603564    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4" podUID="de5114c4-03c6-480b-baea-4f8657ff6a58" containerName="payload-processing" containerID="cri-o://7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457" gracePeriod=25
Oct 30 19:34:26.779968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:26.779926    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"]
Oct 30 19:34:26.780634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:26.780219    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy" containerID="cri-o://106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce" gracePeriod=30
Oct 30 19:34:27.247609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.247581    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:34:27.294135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.292838    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5114c4-03c6-480b-baea-4f8657ff6a58-kube-api-access-dbxmm\" (UniqueName: \"kubernetes.io/projected/de5114c4-03c6-480b-baea-4f8657ff6a58-kube-api-access-dbxmm\") pod \"de5114c4-03c6-480b-baea-4f8657ff6a58\" (UID: \"de5114c4-03c6-480b-baea-4f8657ff6a58\") "
Oct 30 19:34:27.294135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.293012    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5114c4-03c6-480b-baea-4f8657ff6a58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5114c4-03c6-480b-baea-4f8657ff6a58-plugins-config-volume\") pod \"de5114c4-03c6-480b-baea-4f8657ff6a58\" (UID: \"de5114c4-03c6-480b-baea-4f8657ff6a58\") "
Oct 30 19:34:27.299119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.296006    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5114c4-03c6-480b-baea-4f8657ff6a58-plugins-config-volume" pod "de5114c4-03c6-480b-baea-4f8657ff6a58" (UID: "de5114c4-03c6-480b-baea-4f8657ff6a58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:27.321852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.321817    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"]
Oct 30 19:34:27.324931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324575    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1db3bbf8-847c-4004-8b1b-91bca311dbf7" containerName="payload-pre-processing"
Oct 30 19:34:27.324931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324606    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db3bbf8-847c-4004-8b1b-91bca311dbf7" containerName="payload-pre-processing"
Oct 30 19:34:27.324931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324628    6777 cpu_manager.go:414] "containerMap: removing container" podUID="eb1df337-9ecf-43f5-95ad-99faa700a4b0" containerName="payload-processing"
Oct 30 19:34:27.324931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324637    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1df337-9ecf-43f5-95ad-99faa700a4b0" containerName="payload-processing"
Oct 30 19:34:27.324931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324652    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9239a6b1-236b-4e72-a8ae-fabc64d22619" containerName="revoke-keys"
Oct 30 19:34:27.324931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324660    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9239a6b1-236b-4e72-a8ae-fabc64d22619" containerName="revoke-keys"
Oct 30 19:34:27.324931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324696    6777 cpu_manager.go:414] "containerMap: removing container" podUID="de5114c4-03c6-480b-baea-4f8657ff6a58" containerName="payload-processing"
Oct 30 19:34:27.324931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324704    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5114c4-03c6-480b-baea-4f8657ff6a58" containerName="payload-processing"
Oct 30 19:34:27.324931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324932    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db3bbf8-847c-4004-8b1b-91bca311dbf7" containerName="payload-pre-processing"
Oct 30 19:34:27.325626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324949    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1df337-9ecf-43f5-95ad-99faa700a4b0" containerName="payload-processing"
Oct 30 19:34:27.325626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324964    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9239a6b1-236b-4e72-a8ae-fabc64d22619" containerName="revoke-keys"
Oct 30 19:34:27.325626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.324977    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5114c4-03c6-480b-baea-4f8657ff6a58" containerName="payload-processing"
Oct 30 19:34:27.329762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.329728    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5114c4-03c6-480b-baea-4f8657ff6a58-kube-api-access-dbxmm" pod "de5114c4-03c6-480b-baea-4f8657ff6a58" (UID: "de5114c4-03c6-480b-baea-4f8657ff6a58"). InnerVolumeSpecName "kube-api-access-dbxmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:27.330269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.330245    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.339410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.339106    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"]
Oct 30 19:34:27.383314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.383280    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:27.383476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.383425    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:27.397017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.396976    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.397209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.397137    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.397254 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.397217    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l79p2\" (UniqueName: \"kubernetes.io/projected/5fc4c73f-e245-4091-bd82-985ee751f1d3-kube-api-access-l79p2\") pod \"maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.397371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.397355    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5114c4-03c6-480b-baea-4f8657ff6a58-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:27.397414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.397378    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbxmm\" (UniqueName: \"kubernetes.io/projected/de5114c4-03c6-480b-baea-4f8657ff6a58-kube-api-access-dbxmm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:27.498480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.498391    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.498480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.498468    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l79p2\" (UniqueName: \"kubernetes.io/projected/5fc4c73f-e245-4091-bd82-985ee751f1d3-kube-api-access-l79p2\") pod \"maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.498718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.498706    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.499879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.499854    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.501385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.501364    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.512275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.512211    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l79p2\" (UniqueName: \"kubernetes.io/projected/5fc4c73f-e245-4091-bd82-985ee751f1d3-kube-api-access-l79p2\") pod \"maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.630562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.630522    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw" podUID="fef5a740-4ced-428e-bb19-9092a9ae471f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:27.630945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.630522    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp" podUID="cc79b5d0-dbf8-413d-ad18-216d07d3ad08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:27.679827 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.679790    6777 generic.go:356] "Generic (PLEG): container finished" podID="de5114c4-03c6-480b-baea-4f8657ff6a58" containerID="7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457" exitCode=0
Oct 30 19:34:27.680006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.679836    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"
Oct 30 19:34:27.680006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.679876    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4" event={"ID":"de5114c4-03c6-480b-baea-4f8657ff6a58","Type":"ContainerDied","Data":"7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457"}
Oct 30 19:34:27.680006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.679940    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4" event={"ID":"de5114c4-03c6-480b-baea-4f8657ff6a58","Type":"ContainerDied","Data":"8147a907695796fcd835d247877909e6999d48cc23464826365fb38bc47ffb14"}
Oct 30 19:34:27.680006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.679971    6777 scope.go:122] "RemoveContainer" containerID="7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457"
Oct 30 19:34:27.686161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.686139    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:27.692955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.692932    6777 scope.go:122] "RemoveContainer" containerID="7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457"
Oct 30 19:34:27.693324 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:34:27.693303    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457\": container with ID starting with 7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457 not found: ID does not exist" containerID="7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457"
Oct 30 19:34:27.693401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.693332    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457"} err="failed to get container status \"7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457\": rpc error: code = NotFound desc = could not find container \"7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457\": container with ID starting with 7a5e9a1cd626ad712440c19c4cb9b88a1bf723eae514edb1b5ac79768e3ab457 not found: ID does not exist"
Oct 30 19:34:27.715540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.715508    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"]
Oct 30 19:34:27.722762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.722727    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef5dca-659548cc46-g8lr4"]
Oct 30 19:34:27.985232 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:34:27.985200    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc4c73f_e245_4091_bd82_985ee751f1d3.slice/crio-ab7871455c17be11d6e050f9e67bb82d39169bd85108cf50f0efbb69156eba4a WatchSource:0}: Error finding container ab7871455c17be11d6e050f9e67bb82d39169bd85108cf50f0efbb69156eba4a: Status 404 returned error can't find the container with id ab7871455c17be11d6e050f9e67bb82d39169bd85108cf50f0efbb69156eba4a
Oct 30 19:34:27.985833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:27.985810    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"]
Oct 30 19:34:28.117749 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:28.117716    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5114c4-03c6-480b-baea-4f8657ff6a58" path="/var/lib/kubelet/pods/de5114c4-03c6-480b-baea-4f8657ff6a58/volumes"
Oct 30 19:34:28.692798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:28.692701    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc" event={"ID":"5fc4c73f-e245-4091-bd82-985ee751f1d3","Type":"ContainerStarted","Data":"c01821dfbefea79f4ed2c39d29a9a181041c23009305cbb4045768dcee8408b6"}
Oct 30 19:34:28.692798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:28.692768    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc" event={"ID":"5fc4c73f-e245-4091-bd82-985ee751f1d3","Type":"ContainerStarted","Data":"ab7871455c17be11d6e050f9e67bb82d39169bd85108cf50f0efbb69156eba4a"}
Oct 30 19:34:28.716672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:28.716602    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc" podStartSLOduration=1.716584707 podStartE2EDuration="1.716584707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:28.714255337 +0000 UTC m=+1650.711170722" watchObservedRunningTime="2026-10-30 19:34:28.716584707 +0000 UTC m=+1650.713500093"
Oct 30 19:34:29.796093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:29.795029    6777 generic.go:356] "Generic (PLEG): container finished" podID="5fc4c73f-e245-4091-bd82-985ee751f1d3" containerID="c01821dfbefea79f4ed2c39d29a9a181041c23009305cbb4045768dcee8408b6" exitCode=0
Oct 30 19:34:29.796093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:29.795259    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc" event={"ID":"5fc4c73f-e245-4091-bd82-985ee751f1d3","Type":"ContainerDied","Data":"c01821dfbefea79f4ed2c39d29a9a181041c23009305cbb4045768dcee8408b6"}
Oct 30 19:34:30.731640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.730213    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"]
Oct 30 19:34:30.746151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.746096    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:30.763028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.750751    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"]
Oct 30 19:34:30.838743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.838702    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53b781b-mnmdh\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:30.839440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.839404    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53b781b-mnmdh\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:30.839633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.839603    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9vr\" (UniqueName: \"kubernetes.io/projected/94f06436-7a0b-4124-9a73-7430a0d83fcb-kube-api-access-7x9vr\") pod \"maas-api-revoke-sub-a53b781b-mnmdh\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:30.941242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.941193    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9vr\" (UniqueName: \"kubernetes.io/projected/94f06436-7a0b-4124-9a73-7430a0d83fcb-kube-api-access-7x9vr\") pod \"maas-api-revoke-sub-a53b781b-mnmdh\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:30.941453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.941435    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53b781b-mnmdh\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:30.941570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.941505    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53b781b-mnmdh\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:30.942355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.942328    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53b781b-mnmdh\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:30.947719 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.946677    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53b781b-mnmdh\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:30.995443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:30.991780    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9vr\" (UniqueName: \"kubernetes.io/projected/94f06436-7a0b-4124-9a73-7430a0d83fcb-kube-api-access-7x9vr\") pod \"maas-api-revoke-sub-a53b781b-mnmdh\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:31.085155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.085125    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:31.103200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.103164    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:31.144564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.144453    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-openshift-ca\") pod \"5fc4c73f-e245-4091-bd82-985ee751f1d3\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") "
Oct 30 19:34:31.144564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.144509    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-xks-ca\") pod \"5fc4c73f-e245-4091-bd82-985ee751f1d3\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") "
Oct 30 19:34:31.144564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.144542    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc4c73f-e245-4091-bd82-985ee751f1d3-kube-api-access-l79p2\" (UniqueName: \"kubernetes.io/projected/5fc4c73f-e245-4091-bd82-985ee751f1d3-kube-api-access-l79p2\") pod \"5fc4c73f-e245-4091-bd82-985ee751f1d3\" (UID: \"5fc4c73f-e245-4091-bd82-985ee751f1d3\") "
Oct 30 19:34:31.144989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.144856    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-openshift-ca" pod "5fc4c73f-e245-4091-bd82-985ee751f1d3" (UID: "5fc4c73f-e245-4091-bd82-985ee751f1d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:31.145479 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.145454    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:31.147370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.147330    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-xks-ca" pod "5fc4c73f-e245-4091-bd82-985ee751f1d3" (UID: "5fc4c73f-e245-4091-bd82-985ee751f1d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:31.147493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.147438    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc4c73f-e245-4091-bd82-985ee751f1d3-kube-api-access-l79p2" pod "5fc4c73f-e245-4091-bd82-985ee751f1d3" (UID: "5fc4c73f-e245-4091-bd82-985ee751f1d3"). InnerVolumeSpecName "kube-api-access-l79p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:31.247259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.247169    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc4c73f-e245-4091-bd82-985ee751f1d3-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:31.247259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.247203    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l79p2\" (UniqueName: \"kubernetes.io/projected/5fc4c73f-e245-4091-bd82-985ee751f1d3-kube-api-access-l79p2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:31.442666 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:34:31.442634    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f06436_7a0b_4124_9a73_7430a0d83fcb.slice/crio-9ccaded7ee4e0bf0aadfd6888a6b4a7faecb38b4ad389864c4f1b8ee51c94f56 WatchSource:0}: Error finding container 9ccaded7ee4e0bf0aadfd6888a6b4a7faecb38b4ad389864c4f1b8ee51c94f56: Status 404 returned error can't find the container with id 9ccaded7ee4e0bf0aadfd6888a6b4a7faecb38b4ad389864c4f1b8ee51c94f56
Oct 30 19:34:31.444117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.444090    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"]
Oct 30 19:34:31.829008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.828977    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"
Oct 30 19:34:31.829008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.828993    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc" event={"ID":"5fc4c73f-e245-4091-bd82-985ee751f1d3","Type":"ContainerDied","Data":"ab7871455c17be11d6e050f9e67bb82d39169bd85108cf50f0efbb69156eba4a"}
Oct 30 19:34:31.829269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.829045    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab7871455c17be11d6e050f9e67bb82d39169bd85108cf50f0efbb69156eba4a"
Oct 30 19:34:31.830800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:31.830767    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh" event={"ID":"94f06436-7a0b-4124-9a73-7430a0d83fcb","Type":"ContainerStarted","Data":"9ccaded7ee4e0bf0aadfd6888a6b4a7faecb38b4ad389864c4f1b8ee51c94f56"}
Oct 30 19:34:32.277784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.277757    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:34:32.351102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.347894    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-podinfo\") pod \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") "
Oct 30 19:34:32.351102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.348101    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-envoy\") pod \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") "
Oct 30 19:34:32.351102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.348140    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-certs\") pod \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") "
Oct 30 19:34:32.351102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.348251    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-kube-api-access-j77wt\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-kube-api-access-j77wt\") pod \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") "
Oct 30 19:34:32.351102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.348301    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istiod-ca-cert\") pod \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") "
Oct 30 19:34:32.351102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.348338    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-socket\") pod \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") "
Oct 30 19:34:32.351102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.348504    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-token\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-token\") pod \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") "
Oct 30 19:34:32.351102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.348592    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-credential-socket\") pod \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") "
Oct 30 19:34:32.351102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.348680    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-data\") pod \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\" (UID: \"e498f782-d4ee-4bdd-814c-3de1a07e2dc5\") "
Oct 30 19:34:32.367080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.364901    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-socket" pod "e498f782-d4ee-4bdd-814c-3de1a07e2dc5" (UID: "e498f782-d4ee-4bdd-814c-3de1a07e2dc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.367080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.365317    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istiod-ca-cert" pod "e498f782-d4ee-4bdd-814c-3de1a07e2dc5" (UID: "e498f782-d4ee-4bdd-814c-3de1a07e2dc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:32.367080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.365888    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-credential-socket" pod "e498f782-d4ee-4bdd-814c-3de1a07e2dc5" (UID: "e498f782-d4ee-4bdd-814c-3de1a07e2dc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.376204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.370163    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-data" pod "e498f782-d4ee-4bdd-814c-3de1a07e2dc5" (UID: "e498f782-d4ee-4bdd-814c-3de1a07e2dc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.376204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.370954    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.376204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.370981    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.376204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.371007    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.376204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.371021    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.376204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.375504    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-token" pod "e498f782-d4ee-4bdd-814c-3de1a07e2dc5" (UID: "e498f782-d4ee-4bdd-814c-3de1a07e2dc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:32.397385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.389189    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-certs" pod "e498f782-d4ee-4bdd-814c-3de1a07e2dc5" (UID: "e498f782-d4ee-4bdd-814c-3de1a07e2dc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.397385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.396282    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-podinfo" pod "e498f782-d4ee-4bdd-814c-3de1a07e2dc5" (UID: "e498f782-d4ee-4bdd-814c-3de1a07e2dc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:32.404314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.399756    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-kube-api-access-j77wt" pod "e498f782-d4ee-4bdd-814c-3de1a07e2dc5" (UID: "e498f782-d4ee-4bdd-814c-3de1a07e2dc5"). InnerVolumeSpecName "kube-api-access-j77wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:32.404314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.403347    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-envoy" pod "e498f782-d4ee-4bdd-814c-3de1a07e2dc5" (UID: "e498f782-d4ee-4bdd-814c-3de1a07e2dc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.482106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.481956    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.482106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.481999    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.482106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.482015    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.482106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.482033    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j77wt\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-kube-api-access-j77wt\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.482106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.482050    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e498f782-d4ee-4bdd-814c-3de1a07e2dc5-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.625754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.625717    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"]
Oct 30 19:34:32.626385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.626352    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy" containerID="cri-o://26c846b3c6568675dced39f56dd25fbbd24cd4ed0444ac66515be5b86cfddf7a" gracePeriod=30
Oct 30 19:34:32.972370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.972256    6777 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:34:32.972585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:32.972560    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:34:33.112989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.112876    6777 generic.go:356] "Generic (PLEG): container finished" podID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerID="106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce" exitCode=0
Oct 30 19:34:33.113201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.112992    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"
Oct 30 19:34:33.113201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.113017    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" event={"ID":"e498f782-d4ee-4bdd-814c-3de1a07e2dc5","Type":"ContainerDied","Data":"106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce"}
Oct 30 19:34:33.113201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.113113    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd" event={"ID":"e498f782-d4ee-4bdd-814c-3de1a07e2dc5","Type":"ContainerDied","Data":"fe903b8ad4f61fb0dc27815c99c68d801b3993bf8dad797769bafa650f37d756"}
Oct 30 19:34:33.113201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.113142    6777 scope.go:122] "RemoveContainer" containerID="106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce"
Oct 30 19:34:33.116142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.116116    6777 generic.go:356] "Generic (PLEG): container finished" podID="94f06436-7a0b-4124-9a73-7430a0d83fcb" containerID="64c195d9927e1ca00bbbcb10e653a29c9f9c05ef3dbb797a06bc4198d3080388" exitCode=0
Oct 30 19:34:33.116272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.116238    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh" event={"ID":"94f06436-7a0b-4124-9a73-7430a0d83fcb","Type":"ContainerDied","Data":"64c195d9927e1ca00bbbcb10e653a29c9f9c05ef3dbb797a06bc4198d3080388"}
Oct 30 19:34:33.128636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.128612    6777 scope.go:122] "RemoveContainer" containerID="106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce"
Oct 30 19:34:33.129012 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:34:33.128987    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce\": container with ID starting with 106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce not found: ID does not exist" containerID="106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce"
Oct 30 19:34:33.129174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.129022    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce"} err="failed to get container status \"106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce\": rpc error: code = NotFound desc = could not find container \"106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce\": container with ID starting with 106a8a3de6d967d58e5567f0130a4313f8fa0c0c8648b19ee1969bfa1a1376ce not found: ID does not exist"
Oct 30 19:34:33.228405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.228368    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"]
Oct 30 19:34:33.289404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.289010    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-72vxd"]
Oct 30 19:34:33.295983 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.295943    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"]
Oct 30 19:34:33.297957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.297929    6777 cpu_manager.go:414] "containerMap: removing container" podUID="5fc4c73f-e245-4091-bd82-985ee751f1d3" containerName="revoke-keys"
Oct 30 19:34:33.297957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.297958    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc4c73f-e245-4091-bd82-985ee751f1d3" containerName="revoke-keys"
Oct 30 19:34:33.298261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.298023    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy"
Oct 30 19:34:33.298261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.298032    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy"
Oct 30 19:34:33.298362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.298332    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc4c73f-e245-4091-bd82-985ee751f1d3" containerName="revoke-keys"
Oct 30 19:34:33.298415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.298363    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" containerName="istio-proxy"
Oct 30 19:34:33.306360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.306336    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"]
Oct 30 19:34:33.306607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.306458    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.313090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.313046    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1fa500-openshift-default-dockercfg-vvhvs\""
Oct 30 19:34:33.442147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.438934    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-certs\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.442147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.438998    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-token\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.442147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.439043    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-podinfo\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.442147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.439165    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb5014-7d21-4071-a25e-67d977c69bd0-istiod-ca-cert\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.442147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.439295    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.442147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.439344    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfl25\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-kube-api-access-rfl25\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.442147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.439417    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-data\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.442147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.439470    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-envoy\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.442147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.439656    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-credential-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.543080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.541331    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-credential-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.543080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.541487    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-certs\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.543080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.541512    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-token\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.543080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.541550    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-podinfo\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.543080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.541646    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb5014-7d21-4071-a25e-67d977c69bd0-istiod-ca-cert\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.543080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.541763    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.543080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.541810    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfl25\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-kube-api-access-rfl25\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.543080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.541874    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-data\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.543080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.541923    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-envoy\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.555083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.549958    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.555083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.550602    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb5014-7d21-4071-a25e-67d977c69bd0-istiod-ca-cert\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.555083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.551744    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-certs\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.555083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.551986    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-data\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.625082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.571991    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-credential-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.625082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.589022    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfl25\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-kube-api-access-rfl25\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.625082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.589546    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-token\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.625082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.590442    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-podinfo\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.656346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.656309    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-envoy\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:33.924386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:33.924255    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:34.025352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.023578    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"]
Oct 30 19:34:34.031274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.031242    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.043824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.042455    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"]
Oct 30 19:34:34.043824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.043758    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-69c573-openshift-default-dockercfg-5g2t8\""
Oct 30 19:34:34.119704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.119658    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.119973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.119720    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zgm\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-kube-api-access-55zgm\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.119973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.119859    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.120185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.120164    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132771fd-242f-43c0-9c51-32dc14c71f98-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.120279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.120260    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.120420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.120398    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.120583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.120566    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.120742 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.120725    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.120914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.120896    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132771fd-242f-43c0-9c51-32dc14c71f98-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.124924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.124891    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e498f782-d4ee-4bdd-814c-3de1a07e2dc5" path="/var/lib/kubelet/pods/e498f782-d4ee-4bdd-814c-3de1a07e2dc5/volumes"
Oct 30 19:34:34.370197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.368761    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.370197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.368831    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zgm\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-kube-api-access-55zgm\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.370197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.369004    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.370197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.369202    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132771fd-242f-43c0-9c51-32dc14c71f98-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.370197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.369295    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.370197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.369356    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.370197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.369473    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.370197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.369511    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.370197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.369538    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132771fd-242f-43c0-9c51-32dc14c71f98-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.484101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.374927    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.484101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.436455    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.484101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.437152    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132771fd-242f-43c0-9c51-32dc14c71f98-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.484101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.442298    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.503080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.502822    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.633304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.623508    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"]
Oct 30 19:34:34.633304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.623935    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.633304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.624572    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.633304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.625258    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132771fd-242f-43c0-9c51-32dc14c71f98-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.633304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.631762    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zgm\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-kube-api-access-55zgm\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.647603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.646944    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:34.647603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.647074    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:34.647603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.647126    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:34.697497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.697452    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:34.794008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.793977    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:34.927862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.927763    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-openshift-ca\") pod \"94f06436-7a0b-4124-9a73-7430a0d83fcb\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") "
Oct 30 19:34:34.928096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.928052    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-xks-ca\") pod \"94f06436-7a0b-4124-9a73-7430a0d83fcb\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") "
Oct 30 19:34:34.928306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.928286    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f06436-7a0b-4124-9a73-7430a0d83fcb-kube-api-access-7x9vr\" (UniqueName: \"kubernetes.io/projected/94f06436-7a0b-4124-9a73-7430a0d83fcb-kube-api-access-7x9vr\") pod \"94f06436-7a0b-4124-9a73-7430a0d83fcb\" (UID: \"94f06436-7a0b-4124-9a73-7430a0d83fcb\") "
Oct 30 19:34:34.928822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.928794    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-openshift-ca" pod "94f06436-7a0b-4124-9a73-7430a0d83fcb" (UID: "94f06436-7a0b-4124-9a73-7430a0d83fcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:34.929515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.929232    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:34.931866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.931821    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-xks-ca" pod "94f06436-7a0b-4124-9a73-7430a0d83fcb" (UID: "94f06436-7a0b-4124-9a73-7430a0d83fcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:34.932137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:34.932105    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f06436-7a0b-4124-9a73-7430a0d83fcb-kube-api-access-7x9vr" pod "94f06436-7a0b-4124-9a73-7430a0d83fcb" (UID: "94f06436-7a0b-4124-9a73-7430a0d83fcb"). InnerVolumeSpecName "kube-api-access-7x9vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:35.031627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.031593    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f06436-7a0b-4124-9a73-7430a0d83fcb-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:35.031627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.031624    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x9vr\" (UniqueName: \"kubernetes.io/projected/94f06436-7a0b-4124-9a73-7430a0d83fcb-kube-api-access-7x9vr\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:35.042648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.042215    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"]
Oct 30 19:34:35.045458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.045419    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:35.045629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.045607    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:35.045712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.045660    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:35.139960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.139909    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" event={"ID":"08eb5014-7d21-4071-a25e-67d977c69bd0","Type":"ContainerStarted","Data":"c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b"}
Oct 30 19:34:35.140153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.139984    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" event={"ID":"08eb5014-7d21-4071-a25e-67d977c69bd0","Type":"ContainerStarted","Data":"7f262d4119b8d970d4df28c91ac1adaaa438018917a6827d9aa26369775c6fbb"}
Oct 30 19:34:35.141887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.141859    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh" event={"ID":"94f06436-7a0b-4124-9a73-7430a0d83fcb","Type":"ContainerDied","Data":"9ccaded7ee4e0bf0aadfd6888a6b4a7faecb38b4ad389864c4f1b8ee51c94f56"}
Oct 30 19:34:35.142030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.141902    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ccaded7ee4e0bf0aadfd6888a6b4a7faecb38b4ad389864c4f1b8ee51c94f56"
Oct 30 19:34:35.142030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.141943    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"
Oct 30 19:34:35.147098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.145465    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" event={"ID":"132771fd-242f-43c0-9c51-32dc14c71f98","Type":"ContainerStarted","Data":"618fd6a880c63cdee8b7dafb8fbd14283af390a0c61df5a46262036555795f49"}
Oct 30 19:34:35.190382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.190316    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" podStartSLOduration=2.190294078 podStartE2EDuration="2.190294078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:35.181828041 +0000 UTC m=+1657.178743427" watchObservedRunningTime="2026-10-30 19:34:35.190294078 +0000 UTC m=+1657.187209465"
Oct 30 19:34:35.924445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.924408    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:35.931677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:35.931643    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:36.177027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:36.176985    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" event={"ID":"132771fd-242f-43c0-9c51-32dc14c71f98","Type":"ContainerStarted","Data":"a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f"}
Oct 30 19:34:36.177284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:36.177136    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:36.178013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:36.177982    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:34:36.209252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:36.209163    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podStartSLOduration=3.209143173 podStartE2EDuration="3.209143173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:36.202317371 +0000 UTC m=+1658.199232759" watchObservedRunningTime="2026-10-30 19:34:36.209143173 +0000 UTC m=+1658.206058558"
Oct 30 19:34:36.698350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:36.698316    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:36.706006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:36.705978    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:37.184589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:37.184549    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:37.185559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:37.185536    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:34:37.447798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:37.447094    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"]
Oct 30 19:34:37.631467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:37.631428    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw" podUID="fef5a740-4ced-428e-bb19-9092a9ae471f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:37.631652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:37.631556    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp" podUID="cc79b5d0-dbf8-413d-ad18-216d07d3ad08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:38.195193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.195161    6777 generic.go:356] "Generic (PLEG): container finished" podID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerID="26c846b3c6568675dced39f56dd25fbbd24cd4ed0444ac66515be5b86cfddf7a" exitCode=0
Oct 30 19:34:38.195618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.195234    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" event={"ID":"8120b5d3-a576-47cc-a4ff-6b698af54051","Type":"ContainerDied","Data":"26c846b3c6568675dced39f56dd25fbbd24cd4ed0444ac66515be5b86cfddf7a"}
Oct 30 19:34:38.195618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.195293    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s" event={"ID":"8120b5d3-a576-47cc-a4ff-6b698af54051","Type":"ContainerDied","Data":"ac40c09e98fc1b445f7ddd8ea83a2c2232f1736459008eac9c0fbdead4c2574e"}
Oct 30 19:34:38.195618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.195318    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac40c09e98fc1b445f7ddd8ea83a2c2232f1736459008eac9c0fbdead4c2574e"
Oct 30 19:34:38.199578 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.199550    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:34:38.267892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.267855    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-podinfo\") pod \"8120b5d3-a576-47cc-a4ff-6b698af54051\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") "
Oct 30 19:34:38.268139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.267923    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-data\") pod \"8120b5d3-a576-47cc-a4ff-6b698af54051\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") "
Oct 30 19:34:38.268139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.268049    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-envoy\") pod \"8120b5d3-a576-47cc-a4ff-6b698af54051\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") "
Oct 30 19:34:38.268290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.268150    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-socket\") pod \"8120b5d3-a576-47cc-a4ff-6b698af54051\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") "
Oct 30 19:34:38.268290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.268184    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-credential-socket\") pod \"8120b5d3-a576-47cc-a4ff-6b698af54051\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") "
Oct 30 19:34:38.268290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.268282    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8120b5d3-a576-47cc-a4ff-6b698af54051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8120b5d3-a576-47cc-a4ff-6b698af54051-istiod-ca-cert\") pod \"8120b5d3-a576-47cc-a4ff-6b698af54051\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") "
Oct 30 19:34:38.268445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.268369    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-kube-api-access-qkqw9\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-kube-api-access-qkqw9\") pod \"8120b5d3-a576-47cc-a4ff-6b698af54051\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") "
Oct 30 19:34:38.268497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.268484    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-certs\") pod \"8120b5d3-a576-47cc-a4ff-6b698af54051\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") "
Oct 30 19:34:38.268572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.268552    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-token\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-token\") pod \"8120b5d3-a576-47cc-a4ff-6b698af54051\" (UID: \"8120b5d3-a576-47cc-a4ff-6b698af54051\") "
Oct 30 19:34:38.270227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.269827    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-socket" pod "8120b5d3-a576-47cc-a4ff-6b698af54051" (UID: "8120b5d3-a576-47cc-a4ff-6b698af54051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.271616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.271423    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-data" pod "8120b5d3-a576-47cc-a4ff-6b698af54051" (UID: "8120b5d3-a576-47cc-a4ff-6b698af54051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.271616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.271442    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-certs" pod "8120b5d3-a576-47cc-a4ff-6b698af54051" (UID: "8120b5d3-a576-47cc-a4ff-6b698af54051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.271616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.271501    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-credential-socket" pod "8120b5d3-a576-47cc-a4ff-6b698af54051" (UID: "8120b5d3-a576-47cc-a4ff-6b698af54051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.272525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.272255    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8120b5d3-a576-47cc-a4ff-6b698af54051-istiod-ca-cert" pod "8120b5d3-a576-47cc-a4ff-6b698af54051" (UID: "8120b5d3-a576-47cc-a4ff-6b698af54051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:38.275189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.273705    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.275189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.273733    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.275189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.273747    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.275189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.273762    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.275189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.273777    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8120b5d3-a576-47cc-a4ff-6b698af54051-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.278778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.278733    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-envoy" pod "8120b5d3-a576-47cc-a4ff-6b698af54051" (UID: "8120b5d3-a576-47cc-a4ff-6b698af54051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.279081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.279035    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-podinfo" pod "8120b5d3-a576-47cc-a4ff-6b698af54051" (UID: "8120b5d3-a576-47cc-a4ff-6b698af54051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:38.280228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.280201    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-kube-api-access-qkqw9" pod "8120b5d3-a576-47cc-a4ff-6b698af54051" (UID: "8120b5d3-a576-47cc-a4ff-6b698af54051"). InnerVolumeSpecName "kube-api-access-qkqw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:38.281125 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.281047    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-token" pod "8120b5d3-a576-47cc-a4ff-6b698af54051" (UID: "8120b5d3-a576-47cc-a4ff-6b698af54051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:38.376318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.376271    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.376318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.376315    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.376538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.376328    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkqw9\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-kube-api-access-qkqw9\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.376538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:38.376343    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8120b5d3-a576-47cc-a4ff-6b698af54051-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:39.125736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.125650    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:34:39.174489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.173976    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:34:39.193421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.193391    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-plugins-config-volume\") pod \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\" (UID: \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\") "
Oct 30 19:34:39.193788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.193771    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-kube-api-access-ldhrv\" (UniqueName: \"kubernetes.io/projected/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-kube-api-access-ldhrv\") pod \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\" (UID: \"cc79b5d0-dbf8-413d-ad18-216d07d3ad08\") "
Oct 30 19:34:39.193862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.193798    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-plugins-config-volume" pod "cc79b5d0-dbf8-413d-ad18-216d07d3ad08" (UID: "cc79b5d0-dbf8-413d-ad18-216d07d3ad08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:39.194965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.194937    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:39.197923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.197866    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-kube-api-access-ldhrv" pod "cc79b5d0-dbf8-413d-ad18-216d07d3ad08" (UID: "cc79b5d0-dbf8-413d-ad18-216d07d3ad08"). InnerVolumeSpecName "kube-api-access-ldhrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:39.210438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.210407    6777 generic.go:356] "Generic (PLEG): container finished" podID="fef5a740-4ced-428e-bb19-9092a9ae471f" containerID="f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2" exitCode=0
Oct 30 19:34:39.211333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.210470    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"
Oct 30 19:34:39.211333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.210494    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw" event={"ID":"fef5a740-4ced-428e-bb19-9092a9ae471f","Type":"ContainerDied","Data":"f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2"}
Oct 30 19:34:39.211333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.210555    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw" event={"ID":"fef5a740-4ced-428e-bb19-9092a9ae471f","Type":"ContainerDied","Data":"d8b5adea301c82de6a7079fa6ce3e2c962cc1110a51160cbba32b52f33c55868"}
Oct 30 19:34:39.211333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.210585    6777 scope.go:122] "RemoveContainer" containerID="f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2"
Oct 30 19:34:39.212339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.212314    6777 generic.go:356] "Generic (PLEG): container finished" podID="cc79b5d0-dbf8-413d-ad18-216d07d3ad08" containerID="e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5" exitCode=0
Oct 30 19:34:39.212452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.212367    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"
Oct 30 19:34:39.212452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.212400    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp" event={"ID":"cc79b5d0-dbf8-413d-ad18-216d07d3ad08","Type":"ContainerDied","Data":"e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5"}
Oct 30 19:34:39.212569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.212463    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp" event={"ID":"cc79b5d0-dbf8-413d-ad18-216d07d3ad08","Type":"ContainerDied","Data":"53cef52c45a073ae32ca382d3746c316899912749e73d46831c86209ebf16981"}
Oct 30 19:34:39.212735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.212718    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"
Oct 30 19:34:39.245394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.245339    6777 scope.go:122] "RemoveContainer" containerID="f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2"
Oct 30 19:34:39.246733 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:34:39.246707    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2\": container with ID starting with f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2 not found: ID does not exist" containerID="f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2"
Oct 30 19:34:39.246961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.246935    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2"} err="failed to get container status \"f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2\": rpc error: code = NotFound desc = could not find container \"f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2\": container with ID starting with f5b9f84ffadd1ca245156fd9a14ee29ec5a7aae2bf1fef9f489c7ac9155da1c2 not found: ID does not exist"
Oct 30 19:34:39.247181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.247164    6777 scope.go:122] "RemoveContainer" containerID="e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5"
Oct 30 19:34:39.265904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.265881    6777 scope.go:122] "RemoveContainer" containerID="e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5"
Oct 30 19:34:39.266342 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:34:39.266314    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5\": container with ID starting with e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5 not found: ID does not exist" containerID="e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5"
Oct 30 19:34:39.266433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.266355    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5"} err="failed to get container status \"e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5\": rpc error: code = NotFound desc = could not find container \"e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5\": container with ID starting with e47a574c38fd84f320d0d723fdb1e9d1508c2b271d6a758c683c08407ff715e5 not found: ID does not exist"
Oct 30 19:34:39.286108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.286072    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"]
Oct 30 19:34:39.291104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.291075    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"]
Oct 30 19:34:39.291985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.291968    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy"
Oct 30 19:34:39.291985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.291986    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy"
Oct 30 19:34:39.292095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292024    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fef5a740-4ced-428e-bb19-9092a9ae471f" containerName="payload-processing"
Oct 30 19:34:39.292095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292030    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef5a740-4ced-428e-bb19-9092a9ae471f" containerName="payload-processing"
Oct 30 19:34:39.292095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292046    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cc79b5d0-dbf8-413d-ad18-216d07d3ad08" containerName="payload-pre-processing"
Oct 30 19:34:39.292095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292052    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc79b5d0-dbf8-413d-ad18-216d07d3ad08" containerName="payload-pre-processing"
Oct 30 19:34:39.292095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292093    6777 cpu_manager.go:414] "containerMap: removing container" podUID="94f06436-7a0b-4124-9a73-7430a0d83fcb" containerName="revoke-keys"
Oct 30 19:34:39.292095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292099    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f06436-7a0b-4124-9a73-7430a0d83fcb" containerName="revoke-keys"
Oct 30 19:34:39.292386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292223    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc79b5d0-dbf8-413d-ad18-216d07d3ad08" containerName="payload-pre-processing"
Oct 30 19:34:39.292386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292237    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f06436-7a0b-4124-9a73-7430a0d83fcb" containerName="revoke-keys"
Oct 30 19:34:39.292386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292252    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" containerName="istio-proxy"
Oct 30 19:34:39.292386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.292270    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef5a740-4ced-428e-bb19-9092a9ae471f" containerName="payload-processing"
Oct 30 19:34:39.296419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.296389    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef5a740-4ced-428e-bb19-9092a9ae471f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef5a740-4ced-428e-bb19-9092a9ae471f-plugins-config-volume\") pod \"fef5a740-4ced-428e-bb19-9092a9ae471f\" (UID: \"fef5a740-4ced-428e-bb19-9092a9ae471f\") "
Oct 30 19:34:39.296737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.296723    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef5a740-4ced-428e-bb19-9092a9ae471f-kube-api-access-dcw8r\" (UniqueName: \"kubernetes.io/projected/fef5a740-4ced-428e-bb19-9092a9ae471f-kube-api-access-dcw8r\") pod \"fef5a740-4ced-428e-bb19-9092a9ae471f\" (UID: \"fef5a740-4ced-428e-bb19-9092a9ae471f\") "
Oct 30 19:34:39.297311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.297296    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhrv\" (UniqueName: \"kubernetes.io/projected/cc79b5d0-dbf8-413d-ad18-216d07d3ad08-kube-api-access-ldhrv\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:39.297734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.297715    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef5a740-4ced-428e-bb19-9092a9ae471f-plugins-config-volume" pod "fef5a740-4ced-428e-bb19-9092a9ae471f" (UID: "fef5a740-4ced-428e-bb19-9092a9ae471f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:39.300341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.300312    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef5a740-4ced-428e-bb19-9092a9ae471f-kube-api-access-dcw8r" pod "fef5a740-4ced-428e-bb19-9092a9ae471f" (UID: "fef5a740-4ced-428e-bb19-9092a9ae471f"). InnerVolumeSpecName "kube-api-access-dcw8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:39.329007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.328958    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-gbx9s"]
Oct 30 19:34:39.329007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.329014    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"]
Oct 30 19:34:39.329334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.329047    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"]
Oct 30 19:34:39.334748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.334718    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:39.339639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.339611    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"]
Oct 30 19:34:39.339639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.339647    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-33b4d7-5957cfbf6c-trlrp"]
Oct 30 19:34:39.339863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.339666    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"]
Oct 30 19:34:39.339863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.339731    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:39.341816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.341314    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1fa500\""
Oct 30 19:34:39.341816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.341502    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1fa500-dockercfg-52wj8\""
Oct 30 19:34:39.414419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.413473    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9152f3-269a-46bc-b268-7a8173084433-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f\" (UID: \"cb9152f3-269a-46bc-b268-7a8173084433\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:39.414419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.413676    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp\" (UID: \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:39.414419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.413756    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vhf\" (UniqueName: \"kubernetes.io/projected/cb9152f3-269a-46bc-b268-7a8173084433-kube-api-access-w7vhf\") pod \"payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f\" (UID: \"cb9152f3-269a-46bc-b268-7a8173084433\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:39.414419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.413802    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frqh\" (UniqueName: \"kubernetes.io/projected/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-kube-api-access-7frqh\") pod \"payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp\" (UID: \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:39.414419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.413936    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcw8r\" (UniqueName: \"kubernetes.io/projected/fef5a740-4ced-428e-bb19-9092a9ae471f-kube-api-access-dcw8r\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:39.414419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.413953    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef5a740-4ced-428e-bb19-9092a9ae471f-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:39.563443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.536531    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp\" (UID: \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:39.563443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.536646    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vhf\" (UniqueName: \"kubernetes.io/projected/cb9152f3-269a-46bc-b268-7a8173084433-kube-api-access-w7vhf\") pod \"payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f\" (UID: \"cb9152f3-269a-46bc-b268-7a8173084433\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:39.563443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.536700    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frqh\" (UniqueName: \"kubernetes.io/projected/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-kube-api-access-7frqh\") pod \"payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp\" (UID: \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:39.563443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.536892    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9152f3-269a-46bc-b268-7a8173084433-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f\" (UID: \"cb9152f3-269a-46bc-b268-7a8173084433\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:39.638891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.633972    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9152f3-269a-46bc-b268-7a8173084433-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f\" (UID: \"cb9152f3-269a-46bc-b268-7a8173084433\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:39.638891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.638816    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp\" (UID: \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:39.724821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.700753    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frqh\" (UniqueName: \"kubernetes.io/projected/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-kube-api-access-7frqh\") pod \"payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp\" (UID: \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:39.724821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.724471    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vhf\" (UniqueName: \"kubernetes.io/projected/cb9152f3-269a-46bc-b268-7a8173084433-kube-api-access-w7vhf\") pod \"payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f\" (UID: \"cb9152f3-269a-46bc-b268-7a8173084433\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:39.780315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.780281    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"]
Oct 30 19:34:39.783469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.783442    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:39.784494 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.784473    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-33b4d7-77445cb55d-h2jzw"]
Oct 30 19:34:39.884447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.884003    6777 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:34:39.884447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.884122    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:39.938579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:39.938267    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:40.102289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.102238    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"]
Oct 30 19:34:40.126648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.126611    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8120b5d3-a576-47cc-a4ff-6b698af54051" path="/var/lib/kubelet/pods/8120b5d3-a576-47cc-a4ff-6b698af54051/volumes"
Oct 30 19:34:40.127285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.127263    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc79b5d0-dbf8-413d-ad18-216d07d3ad08" path="/var/lib/kubelet/pods/cc79b5d0-dbf8-413d-ad18-216d07d3ad08/volumes"
Oct 30 19:34:40.127737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.127720    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef5a740-4ced-428e-bb19-9092a9ae471f" path="/var/lib/kubelet/pods/fef5a740-4ced-428e-bb19-9092a9ae471f/volumes"
Oct 30 19:34:40.243352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.243316    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp" event={"ID":"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb","Type":"ContainerStarted","Data":"95d278e907d14cee1618592a72d79281f4831820d51b6d373b310764125e969f"}
Oct 30 19:34:40.272016 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.271986    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"]
Oct 30 19:34:40.300856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.300818    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"]
Oct 30 19:34:40.333349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.333309    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"]
Oct 30 19:34:40.333567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.333441    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:40.338040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.338017    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-69c573\""
Oct 30 19:34:40.379244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.379081    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"]
Oct 30 19:34:40.391365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.391340    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:40.397853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.397827    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-69c573-dockercfg-scqkg\""
Oct 30 19:34:40.402492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.402231    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"]
Oct 30 19:34:40.768450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.768409    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt\" (UID: \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:40.768787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.768490    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f8ec32-8902-466d-89d2-bb1b83baf1f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s\" (UID: \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:40.768787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.768533    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rs9\" (UniqueName: \"kubernetes.io/projected/09f8ec32-8902-466d-89d2-bb1b83baf1f6-kube-api-access-56rs9\") pod \"payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s\" (UID: \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:40.768921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.768875    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdh6\" (UniqueName: \"kubernetes.io/projected/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-kube-api-access-jhdh6\") pod \"payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt\" (UID: \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:40.871326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.871279    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdh6\" (UniqueName: \"kubernetes.io/projected/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-kube-api-access-jhdh6\") pod \"payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt\" (UID: \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:40.871527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.871507    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt\" (UID: \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:40.871618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.871559    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f8ec32-8902-466d-89d2-bb1b83baf1f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s\" (UID: \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:40.871618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.871603    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rs9\" (UniqueName: \"kubernetes.io/projected/09f8ec32-8902-466d-89d2-bb1b83baf1f6-kube-api-access-56rs9\") pod \"payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s\" (UID: \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:40.872550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.872523    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt\" (UID: \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:40.872942 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.872915    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f8ec32-8902-466d-89d2-bb1b83baf1f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s\" (UID: \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:40.885988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.885956    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdh6\" (UniqueName: \"kubernetes.io/projected/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-kube-api-access-jhdh6\") pod \"payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt\" (UID: \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:40.886200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.885989    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rs9\" (UniqueName: \"kubernetes.io/projected/09f8ec32-8902-466d-89d2-bb1b83baf1f6-kube-api-access-56rs9\") pod \"payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s\" (UID: \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:40.973957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:40.973919    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:41.013761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.013732    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:41.259699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.259650    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp" event={"ID":"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb","Type":"ContainerStarted","Data":"5ebc2f7b4f64e0718dcc4b4a524702cee1f0cd0c438c535376376569032d348d"}
Oct 30 19:34:41.260153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.259754    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:41.262371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.262337    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f" event={"ID":"cb9152f3-269a-46bc-b268-7a8173084433","Type":"ContainerStarted","Data":"43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21"}
Oct 30 19:34:41.262520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.262400    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f" event={"ID":"cb9152f3-269a-46bc-b268-7a8173084433","Type":"ContainerStarted","Data":"238cf0545631798729a65d935ef2eb0658983cffde3c308fc50046caa2be438b"}
Oct 30 19:34:41.262520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.262452    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:41.282430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.282372    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp" podStartSLOduration=2.282357094 podStartE2EDuration="2.282357094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:41.279994583 +0000 UTC m=+1663.276909969" watchObservedRunningTime="2026-10-30 19:34:41.282357094 +0000 UTC m=+1663.279272479"
Oct 30 19:34:41.301540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.301489    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f" podStartSLOduration=2.301475335 podStartE2EDuration="2.301475335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:41.300317721 +0000 UTC m=+1663.297233107" watchObservedRunningTime="2026-10-30 19:34:41.301475335 +0000 UTC m=+1663.298390724"
Oct 30 19:34:41.318434 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:34:41.318388    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecabcbae_0ee0_45f9_8b89_54052c9a5a65.slice/crio-b902815daf0feec7c114be3aed08816b22ef1363b11b55bbed24ef994dd22d10 WatchSource:0}: Error finding container b902815daf0feec7c114be3aed08816b22ef1363b11b55bbed24ef994dd22d10: Status 404 returned error can't find the container with id b902815daf0feec7c114be3aed08816b22ef1363b11b55bbed24ef994dd22d10
Oct 30 19:34:41.319384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.319363    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"]
Oct 30 19:34:41.359710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:41.359680    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"]
Oct 30 19:34:41.361715 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:34:41.361690    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f8ec32_8902_466d_89d2_bb1b83baf1f6.slice/crio-0fbc95182c0a4b42f901b3c860adcedeac190584f78692e685213147198b9772 WatchSource:0}: Error finding container 0fbc95182c0a4b42f901b3c860adcedeac190584f78692e685213147198b9772: Status 404 returned error can't find the container with id 0fbc95182c0a4b42f901b3c860adcedeac190584f78692e685213147198b9772
Oct 30 19:34:42.273975 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:42.273937    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt" event={"ID":"ecabcbae-0ee0-45f9-8b89-54052c9a5a65","Type":"ContainerStarted","Data":"9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b"}
Oct 30 19:34:42.275401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:42.274002    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt" event={"ID":"ecabcbae-0ee0-45f9-8b89-54052c9a5a65","Type":"ContainerStarted","Data":"b902815daf0feec7c114be3aed08816b22ef1363b11b55bbed24ef994dd22d10"}
Oct 30 19:34:42.275401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:42.274113    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:42.276223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:42.276187    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s" event={"ID":"09f8ec32-8902-466d-89d2-bb1b83baf1f6","Type":"ContainerStarted","Data":"a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c"}
Oct 30 19:34:42.276339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:42.276240    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s" event={"ID":"09f8ec32-8902-466d-89d2-bb1b83baf1f6","Type":"ContainerStarted","Data":"0fbc95182c0a4b42f901b3c860adcedeac190584f78692e685213147198b9772"}
Oct 30 19:34:42.297401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:42.297336    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt" podStartSLOduration=2.29731799 podStartE2EDuration="2.29731799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:42.294922961 +0000 UTC m=+1664.291838338" watchObservedRunningTime="2026-10-30 19:34:42.29731799 +0000 UTC m=+1664.294233376"
Oct 30 19:34:42.314611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:42.314552    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s" podStartSLOduration=2.314536634 podStartE2EDuration="2.314536634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:42.312762523 +0000 UTC m=+1664.309677912" watchObservedRunningTime="2026-10-30 19:34:42.314536634 +0000 UTC m=+1664.311452018"
Oct 30 19:34:42.451617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:42.451512    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg" podUID="950eeb12-a96d-4fac-b7c2-091cd8fa93c2" containerName="payload-pre-processing" containerID="cri-o://f29a1b6b4dbf891d9a41fdbe9666fc66ae4964d84de5d91a4a335a7147f216aa" gracePeriod=25
Oct 30 19:34:43.290307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:43.290271    6777 generic.go:356] "Generic (PLEG): container finished" podID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerID="2fdc81f7cc88d01643eda59c2118cff4ba5f8101223e33927ec67bc91ddfc760" exitCode=0
Oct 30 19:34:43.290756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:43.290361    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" event={"ID":"c13df2c0-4690-4548-bf78-a4205a6cae13","Type":"ContainerDied","Data":"2fdc81f7cc88d01643eda59c2118cff4ba5f8101223e33927ec67bc91ddfc760"}
Oct 30 19:34:43.290756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:43.290610    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:43.291462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:43.291440    6777 scope.go:122] "RemoveContainer" containerID="2fdc81f7cc88d01643eda59c2118cff4ba5f8101223e33927ec67bc91ddfc760"
Oct 30 19:34:43.293641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:43.293605    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:43.293731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:43.293700    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:43.293776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:43.293748    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:34:43.646135 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:34:43.645972    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:34:43.770830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:43.770789    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg" podUID="950eeb12-a96d-4fac-b7c2-091cd8fa93c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:44.319081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:44.318113    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" event={"ID":"c13df2c0-4690-4548-bf78-a4205a6cae13","Type":"ContainerStarted","Data":"4c0cf75fc8bf0861914344452415199463b7811f7d4d4ac21e3a16dc1b273ef5"}
Oct 30 19:34:44.967070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:44.966905    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:34:44.969881 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:44.969812    6777 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:34:44.970132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:44.969878    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 19:34:45.591195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:45.591026    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"]
Oct 30 19:34:45.770146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:45.759953    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:45.880292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:45.767895    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:45.971328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:45.971271    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:46.115284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.115220    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"]
Oct 30 19:34:46.123602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.123565    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.182472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.182365    6777 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:34:46.182755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.182712    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 19:34:46.216390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.212462    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:46.216685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.212831    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:46.300537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.300499    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"]
Oct 30 19:34:46.334014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.331676    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.334014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.331839    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grpmr\" (UniqueName: \"kubernetes.io/projected/76a8e66c-42d3-44bf-b49b-2450713eb933-kube-api-access-grpmr\") pod \"maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.334014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.332163    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.433822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.433712    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.433822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.433818    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpmr\" (UniqueName: \"kubernetes.io/projected/76a8e66c-42d3-44bf-b49b-2450713eb933-kube-api-access-grpmr\") pod \"maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.438169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.437438    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.438169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.437987    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.438793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.438770    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.447603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.447573    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpmr\" (UniqueName: \"kubernetes.io/projected/76a8e66c-42d3-44bf-b49b-2450713eb933-kube-api-access-grpmr\") pod \"maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.600024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.599982    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:46.929239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.920234    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"]
Oct 30 19:34:46.980143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.977619    6777 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:34:46.980321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:46.980160    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 19:34:47.126753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:47.126718    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg" event={"ID":"76a8e66c-42d3-44bf-b49b-2450713eb933","Type":"ContainerStarted","Data":"856dda271a14853e3c0a12e4808bb63b68a89854e654df3f1b719f8411e65d80"}
Oct 30 19:34:47.979513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:47.979474    6777 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:34:47.980117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:47.980075    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 19:34:48.503552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:48.483268    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg" event={"ID":"76a8e66c-42d3-44bf-b49b-2450713eb933","Type":"ContainerStarted","Data":"1f9897283ed010cf16f7cae3623ce7e88b4bcd237f9bbb5a926da91ca285eeff"}
Oct 30 19:34:48.535021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:48.534979    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"]
Oct 30 19:34:48.542120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:48.541800    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afdf16-xrxpc"]
Oct 30 19:34:48.974667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:48.974563    6777 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:34:48.974667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:48.974627    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 19:34:49.500574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.500519    6777 generic.go:356] "Generic (PLEG): container finished" podID="76a8e66c-42d3-44bf-b49b-2450713eb933" containerID="1f9897283ed010cf16f7cae3623ce7e88b4bcd237f9bbb5a926da91ca285eeff" exitCode=0
Oct 30 19:34:49.501773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.501120    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg" event={"ID":"76a8e66c-42d3-44bf-b49b-2450713eb933","Type":"ContainerDied","Data":"1f9897283ed010cf16f7cae3623ce7e88b4bcd237f9bbb5a926da91ca285eeff"}
Oct 30 19:34:49.844371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.844341    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:49.974182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.973296    6777 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:34:49.974182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.973365    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 19:34:49.998790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.998749    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-xks-ca\") pod \"76a8e66c-42d3-44bf-b49b-2450713eb933\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") "
Oct 30 19:34:49.999000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.998973    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a8e66c-42d3-44bf-b49b-2450713eb933-kube-api-access-grpmr\" (UniqueName: \"kubernetes.io/projected/76a8e66c-42d3-44bf-b49b-2450713eb933-kube-api-access-grpmr\") pod \"76a8e66c-42d3-44bf-b49b-2450713eb933\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") "
Oct 30 19:34:49.999089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.999050    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-openshift-ca\") pod \"76a8e66c-42d3-44bf-b49b-2450713eb933\" (UID: \"76a8e66c-42d3-44bf-b49b-2450713eb933\") "
Oct 30 19:34:49.999507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.999429    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-openshift-ca" pod "76a8e66c-42d3-44bf-b49b-2450713eb933" (UID: "76a8e66c-42d3-44bf-b49b-2450713eb933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:49.999941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:49.999918    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:50.000918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.000886    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-xks-ca" pod "76a8e66c-42d3-44bf-b49b-2450713eb933" (UID: "76a8e66c-42d3-44bf-b49b-2450713eb933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:50.001235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.001215    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a8e66c-42d3-44bf-b49b-2450713eb933-kube-api-access-grpmr" pod "76a8e66c-42d3-44bf-b49b-2450713eb933" (UID: "76a8e66c-42d3-44bf-b49b-2450713eb933"). InnerVolumeSpecName "kube-api-access-grpmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:50.102158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.102072    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a8e66c-42d3-44bf-b49b-2450713eb933-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:50.102158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.102105    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grpmr\" (UniqueName: \"kubernetes.io/projected/76a8e66c-42d3-44bf-b49b-2450713eb933-kube-api-access-grpmr\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:50.115808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.115764    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc4c73f-e245-4091-bd82-985ee751f1d3" path="/var/lib/kubelet/pods/5fc4c73f-e245-4091-bd82-985ee751f1d3/volumes"
Oct 30 19:34:50.512696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.512616    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"
Oct 30 19:34:50.512696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.512648    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg" event={"ID":"76a8e66c-42d3-44bf-b49b-2450713eb933","Type":"ContainerDied","Data":"856dda271a14853e3c0a12e4808bb63b68a89854e654df3f1b719f8411e65d80"}
Oct 30 19:34:50.513135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.512702    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="856dda271a14853e3c0a12e4808bb63b68a89854e654df3f1b719f8411e65d80"
Oct 30 19:34:50.964626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.964594    6777 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:34:50.964829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:50.964650    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 19:34:51.381555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:51.381517    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"]
Oct 30 19:34:51.381814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:51.381791    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy" containerID="cri-o://d87dd1c13722607b0c066f4101923be25b61449ad51178f3f1210fe8364a7f1f" gracePeriod=30
Oct 30 19:34:51.971194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:51.971164    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:34:51.971685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:51.971639    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:34:51.972453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:51.972430    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:34:52.279235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:52.279209    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:34:52.279387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:52.279289    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:34:53.294413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:53.294384    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:34:53.772660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:53.772573    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg" podUID="950eeb12-a96d-4fac-b7c2-091cd8fa93c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:54.321421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:54.321391    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:34:55.423493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:55.423455    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"]
Oct 30 19:34:56.010102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.010026    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"]
Oct 30 19:34:56.599109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.599011    6777 generic.go:356] "Generic (PLEG): container finished" podID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerID="d87dd1c13722607b0c066f4101923be25b61449ad51178f3f1210fe8364a7f1f" exitCode=0
Oct 30 19:34:56.599620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.599138    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" event={"ID":"16a0608a-109b-4aef-88ab-52b53be4ca45","Type":"ContainerDied","Data":"d87dd1c13722607b0c066f4101923be25b61449ad51178f3f1210fe8364a7f1f"}
Oct 30 19:34:56.599620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.599216    6777 scope.go:122] "RemoveContainer" containerID="6f3ba36c1d6b8eab6d00285847848023a7d77100a36f1e4090a27b88a6b7d0b4"
Oct 30 19:34:56.829886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.829858    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:34:56.875577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.875538    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-envoy\") pod \"16a0608a-109b-4aef-88ab-52b53be4ca45\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") "
Oct 30 19:34:56.875750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.875609    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-token\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-token\") pod \"16a0608a-109b-4aef-88ab-52b53be4ca45\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") "
Oct 30 19:34:56.875750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.875741    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-podinfo\") pod \"16a0608a-109b-4aef-88ab-52b53be4ca45\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") "
Oct 30 19:34:56.875849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.875786    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-credential-socket\") pod \"16a0608a-109b-4aef-88ab-52b53be4ca45\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") "
Oct 30 19:34:56.875953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.875932    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-kube-api-access-9hg84\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-kube-api-access-9hg84\") pod \"16a0608a-109b-4aef-88ab-52b53be4ca45\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") "
Oct 30 19:34:56.876430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.876138    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-credential-socket" pod "16a0608a-109b-4aef-88ab-52b53be4ca45" (UID: "16a0608a-109b-4aef-88ab-52b53be4ca45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:56.876430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.876294    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-data\") pod \"16a0608a-109b-4aef-88ab-52b53be4ca45\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") "
Oct 30 19:34:56.876430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.876369    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-socket\") pod \"16a0608a-109b-4aef-88ab-52b53be4ca45\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") "
Oct 30 19:34:56.876430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.876425    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-certs\") pod \"16a0608a-109b-4aef-88ab-52b53be4ca45\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") "
Oct 30 19:34:56.876830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.876613    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a0608a-109b-4aef-88ab-52b53be4ca45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a0608a-109b-4aef-88ab-52b53be4ca45-istiod-ca-cert\") pod \"16a0608a-109b-4aef-88ab-52b53be4ca45\" (UID: \"16a0608a-109b-4aef-88ab-52b53be4ca45\") "
Oct 30 19:34:56.878790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.877411    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-socket" pod "16a0608a-109b-4aef-88ab-52b53be4ca45" (UID: "16a0608a-109b-4aef-88ab-52b53be4ca45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:56.878790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.877593    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-data" pod "16a0608a-109b-4aef-88ab-52b53be4ca45" (UID: "16a0608a-109b-4aef-88ab-52b53be4ca45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:56.878790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.877711    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-certs" pod "16a0608a-109b-4aef-88ab-52b53be4ca45" (UID: "16a0608a-109b-4aef-88ab-52b53be4ca45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:56.878790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.877790    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a0608a-109b-4aef-88ab-52b53be4ca45-istiod-ca-cert" pod "16a0608a-109b-4aef-88ab-52b53be4ca45" (UID: "16a0608a-109b-4aef-88ab-52b53be4ca45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:56.878790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.878737    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:56.878790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.878763    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:56.878790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.878777    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:56.878790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.878791    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:56.879161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.878805    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a0608a-109b-4aef-88ab-52b53be4ca45-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:56.879161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.878930    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-podinfo" pod "16a0608a-109b-4aef-88ab-52b53be4ca45" (UID: "16a0608a-109b-4aef-88ab-52b53be4ca45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:56.881121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.881051    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-kube-api-access-9hg84" pod "16a0608a-109b-4aef-88ab-52b53be4ca45" (UID: "16a0608a-109b-4aef-88ab-52b53be4ca45"). InnerVolumeSpecName "kube-api-access-9hg84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:56.882813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.882150    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-envoy" pod "16a0608a-109b-4aef-88ab-52b53be4ca45" (UID: "16a0608a-109b-4aef-88ab-52b53be4ca45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:56.883704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.883681    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-token" pod "16a0608a-109b-4aef-88ab-52b53be4ca45" (UID: "16a0608a-109b-4aef-88ab-52b53be4ca45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:56.980539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.980502    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hg84\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-kube-api-access-9hg84\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:56.980539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.980539    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:56.980768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.980552    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:56.980768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:56.980565    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a0608a-109b-4aef-88ab-52b53be4ca45-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.474628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.474593    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"]
Oct 30 19:34:57.475565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.475541    6777 cpu_manager.go:414] "containerMap: removing container" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy"
Oct 30 19:34:57.475565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.475561    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy"
Oct 30 19:34:57.475816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.475590    6777 cpu_manager.go:414] "containerMap: removing container" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy"
Oct 30 19:34:57.475816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.475596    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy"
Oct 30 19:34:57.475816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.475614    6777 cpu_manager.go:414] "containerMap: removing container" podUID="76a8e66c-42d3-44bf-b49b-2450713eb933" containerName="revoke-keys"
Oct 30 19:34:57.475816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.475619    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a8e66c-42d3-44bf-b49b-2450713eb933" containerName="revoke-keys"
Oct 30 19:34:57.475816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.475813    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy"
Oct 30 19:34:57.476723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.475835    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a8e66c-42d3-44bf-b49b-2450713eb933" containerName="revoke-keys"
Oct 30 19:34:57.486769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.485661    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:57.494922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.494812    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:57.502304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.502281    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:57.510677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.510643    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"]
Oct 30 19:34:57.589202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.589160    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"]
Oct 30 19:34:57.590758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.590721    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy" containerID="cri-o://89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709" gracePeriod=30
Oct 30 19:34:57.616174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.616121    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"
Oct 30 19:34:57.619140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.616365    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b" event={"ID":"16a0608a-109b-4aef-88ab-52b53be4ca45","Type":"ContainerDied","Data":"d157cf5285e642a298c3f1de3ce3d48a837e647881db0a7e57dc9550f5d25b5d"}
Oct 30 19:34:57.619298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.619183    6777 scope.go:122] "RemoveContainer" containerID="d87dd1c13722607b0c066f4101923be25b61449ad51178f3f1210fe8364a7f1f"
Oct 30 19:34:57.654807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.654771    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"]
Oct 30 19:34:57.659262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.659236    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-bgt7b"]
Oct 30 19:34:57.745499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.741291    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c2a3340-szs9j\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:57.745499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.741390    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqlv\" (UniqueName: \"kubernetes.io/projected/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-kube-api-access-bfqlv\") pod \"maas-api-revoke-sub-3c2a3340-szs9j\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:57.745499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.741456    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c2a3340-szs9j\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:57.876117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.874413    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c2a3340-szs9j\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:57.876117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.874518    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqlv\" (UniqueName: \"kubernetes.io/projected/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-kube-api-access-bfqlv\") pod \"maas-api-revoke-sub-3c2a3340-szs9j\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:57.876117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.874579    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c2a3340-szs9j\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:57.876117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.875582    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c2a3340-szs9j\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:57.980562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:57.980495    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c2a3340-szs9j\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:58.013150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:58.011849    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqlv\" (UniqueName: \"kubernetes.io/projected/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-kube-api-access-bfqlv\") pod \"maas-api-revoke-sub-3c2a3340-szs9j\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:58.120314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:58.120241    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" path="/var/lib/kubelet/pods/16a0608a-109b-4aef-88ab-52b53be4ca45/volumes"
Oct 30 19:34:58.121333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:58.121308    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:58.128546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:58.128523    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:34:58.430279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:58.430250    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"]
Oct 30 19:34:58.431164 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:34:58.431138    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc4cfcb_6fe1_4c3d_9e99_5f935a8e7a0a.slice/crio-2c9aae9806db8587d877e76d3f8e564857b0ade1ca5d214b23f20d5fb23d30eb WatchSource:0}: Error finding container 2c9aae9806db8587d877e76d3f8e564857b0ade1ca5d214b23f20d5fb23d30eb: Status 404 returned error can't find the container with id 2c9aae9806db8587d877e76d3f8e564857b0ade1ca5d214b23f20d5fb23d30eb
Oct 30 19:34:58.628773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:58.628737    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j" event={"ID":"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a","Type":"ContainerStarted","Data":"2c9aae9806db8587d877e76d3f8e564857b0ade1ca5d214b23f20d5fb23d30eb"}
Oct 30 19:34:59.641002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:59.640959    6777 generic.go:356] "Generic (PLEG): container finished" podID="5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a" containerID="e15f2ad651a81e92256339292512b5f732e3c735eaee6b46fafaf8b059929865" exitCode=0
Oct 30 19:34:59.641542 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:59.641042    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j" event={"ID":"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a","Type":"ContainerDied","Data":"e15f2ad651a81e92256339292512b5f732e3c735eaee6b46fafaf8b059929865"}
Oct 30 19:34:59.731345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:34:59.731304    6777 scope.go:122] "RemoveContainer" containerID="3b6841fb24347d71448857bb4d299ed136295b515864c59adbc4f962f6970a72"
Oct 30 19:35:00.424561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:00.424517    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j" podUID="7d475743-3ee9-452a-bd4c-6de427d0a778" containerName="payload-pre-processing" containerID="cri-o://c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747" gracePeriod=25
Oct 30 19:35:00.739263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:00.739220    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j" podUID="7d475743-3ee9-452a-bd4c-6de427d0a778" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:00.854740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:00.854710    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:35:00.986145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:00.976300    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-kube-api-access-bfqlv\" (UniqueName: \"kubernetes.io/projected/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-kube-api-access-bfqlv\") pod \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") "
Oct 30 19:35:00.986145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:00.976519    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-xks-ca\") pod \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") "
Oct 30 19:35:00.986145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:00.976822    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-openshift-ca\") pod \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\" (UID: \"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a\") "
Oct 30 19:35:01.014334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.013514    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh" podUID="1d962588-5797-46dd-9413-e4c2efcf21f8" containerName="payload-pre-processing" containerID="cri-o://1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479" gracePeriod=25
Oct 30 19:35:01.026388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.023282    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-openshift-ca" pod "5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a" (UID: "5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:01.119152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.111488    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.119152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.113580    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-xks-ca" pod "5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a" (UID: "5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:01.119152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.119122    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-kube-api-access-bfqlv" pod "5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a" (UID: "5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a"). InnerVolumeSpecName "kube-api-access-bfqlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:01.119470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.119282    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx" podUID="f07f718a-19bb-4e4c-a4c4-f6a10b687225" containerName="main" containerID="cri-o://b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663" gracePeriod=45
Oct 30 19:35:01.213329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.213296    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfqlv\" (UniqueName: \"kubernetes.io/projected/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-kube-api-access-bfqlv\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.213329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.213327    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.422116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.422048    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:35:01.517275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.517190    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-model-cache\") pod \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") "
Oct 30 19:35:01.517435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.517414    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-home\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-home\") pod \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") "
Oct 30 19:35:01.517478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.517438    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-dshm\") pod \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") "
Oct 30 19:35:01.517478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.517447    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-model-cache" pod "f07f718a-19bb-4e4c-a4c4-f6a10b687225" (UID: "f07f718a-19bb-4e4c-a4c4-f6a10b687225"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:01.517541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.517516    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07f718a-19bb-4e4c-a4c4-f6a10b687225-kube-api-access-pj4px\" (UniqueName: \"kubernetes.io/projected/f07f718a-19bb-4e4c-a4c4-f6a10b687225-kube-api-access-pj4px\") pod \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") "
Oct 30 19:35:01.517658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.517645    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tmp-dir\") pod \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") "
Oct 30 19:35:01.517712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.517652    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-home" pod "f07f718a-19bb-4e4c-a4c4-f6a10b687225" (UID: "f07f718a-19bb-4e4c-a4c4-f6a10b687225"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:01.517712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.517667    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tls-certs\" (UniqueName: \"kubernetes.io/secret/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tls-certs\") pod \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\" (UID: \"f07f718a-19bb-4e4c-a4c4-f6a10b687225\") "
Oct 30 19:35:01.517921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.517898    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tmp-dir" pod "f07f718a-19bb-4e4c-a4c4-f6a10b687225" (UID: "f07f718a-19bb-4e4c-a4c4-f6a10b687225"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:01.518502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.518480    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.518591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.518509    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.518591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.518528    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.519945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.519924    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tls-certs" pod "f07f718a-19bb-4e4c-a4c4-f6a10b687225" (UID: "f07f718a-19bb-4e4c-a4c4-f6a10b687225"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:01.520025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.519928    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07f718a-19bb-4e4c-a4c4-f6a10b687225-kube-api-access-pj4px" pod "f07f718a-19bb-4e4c-a4c4-f6a10b687225" (UID: "f07f718a-19bb-4e4c-a4c4-f6a10b687225"). InnerVolumeSpecName "kube-api-access-pj4px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:01.520025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.519971    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-dshm" pod "f07f718a-19bb-4e4c-a4c4-f6a10b687225" (UID: "f07f718a-19bb-4e4c-a4c4-f6a10b687225"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:01.621842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.620737    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f07f718a-19bb-4e4c-a4c4-f6a10b687225-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.621842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.620781    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f07f718a-19bb-4e4c-a4c4-f6a10b687225-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.621842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.620796    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4px\" (UniqueName: \"kubernetes.io/projected/f07f718a-19bb-4e4c-a4c4-f6a10b687225-kube-api-access-pj4px\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.662071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.662024    6777 generic.go:356] "Generic (PLEG): container finished" podID="f07f718a-19bb-4e4c-a4c4-f6a10b687225" containerID="b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663" exitCode=0
Oct 30 19:35:01.662238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.662084    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"
Oct 30 19:35:01.662238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.662152    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx" event={"ID":"f07f718a-19bb-4e4c-a4c4-f6a10b687225","Type":"ContainerDied","Data":"b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663"}
Oct 30 19:35:01.662238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.662195    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx" event={"ID":"f07f718a-19bb-4e4c-a4c4-f6a10b687225","Type":"ContainerDied","Data":"ee9a1f2bd6ead8bf98c036fe6ddd6d4024d72e5f749812f3171f07f2413a7126"}
Oct 30 19:35:01.662238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.662211    6777 scope.go:122] "RemoveContainer" containerID="b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663"
Oct 30 19:35:01.664222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.664199    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j" event={"ID":"5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a","Type":"ContainerDied","Data":"2c9aae9806db8587d877e76d3f8e564857b0ade1ca5d214b23f20d5fb23d30eb"}
Oct 30 19:35:01.664328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.664238    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c9aae9806db8587d877e76d3f8e564857b0ade1ca5d214b23f20d5fb23d30eb"
Oct 30 19:35:01.664328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.664241    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"
Oct 30 19:35:01.679343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.679231    6777 scope.go:122] "RemoveContainer" containerID="b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663"
Oct 30 19:35:01.679755 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:01.679714    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663\": container with ID starting with b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663 not found: ID does not exist" containerID="b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663"
Oct 30 19:35:01.679890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.679760    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663"} err="failed to get container status \"b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663\": rpc error: code = NotFound desc = could not find container \"b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663\": container with ID starting with b5cf47aa57f5fe3cc4f1ffef5a53a41e0d8bc8fa568da3a6b9653069f0e05663 not found: ID does not exist"
Oct 30 19:35:01.697575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.697537    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"]
Oct 30 19:35:01.701887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:01.701864    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-69f3a9/test-model-69f3a9-kserve-75cf4cc887-k8qnx"]
Oct 30 19:35:02.116429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.116347    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07f718a-19bb-4e4c-a4c4-f6a10b687225" path="/var/lib/kubelet/pods/f07f718a-19bb-4e4c-a4c4-f6a10b687225/volumes"
Oct 30 19:35:02.686713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.682483    6777 generic.go:356] "Generic (PLEG): container finished" podID="950eeb12-a96d-4fac-b7c2-091cd8fa93c2" containerID="f29a1b6b4dbf891d9a41fdbe9666fc66ae4964d84de5d91a4a335a7147f216aa" exitCode=0
Oct 30 19:35:02.686713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.682650    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg" event={"ID":"950eeb12-a96d-4fac-b7c2-091cd8fa93c2","Type":"ContainerDied","Data":"f29a1b6b4dbf891d9a41fdbe9666fc66ae4964d84de5d91a4a335a7147f216aa"}
Oct 30 19:35:02.730895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.730839    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:35:02.856981 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.856946    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-kube-api-access-f9rps\" (UniqueName: \"kubernetes.io/projected/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-kube-api-access-f9rps\") pod \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\" (UID: \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\") "
Oct 30 19:35:02.857244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.857112    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-plugins-config-volume\") pod \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\" (UID: \"950eeb12-a96d-4fac-b7c2-091cd8fa93c2\") "
Oct 30 19:35:02.857553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.857526    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-plugins-config-volume" pod "950eeb12-a96d-4fac-b7c2-091cd8fa93c2" (UID: "950eeb12-a96d-4fac-b7c2-091cd8fa93c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:02.857885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.857868    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:02.870127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.860766    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-kube-api-access-f9rps" pod "950eeb12-a96d-4fac-b7c2-091cd8fa93c2" (UID: "950eeb12-a96d-4fac-b7c2-091cd8fa93c2"). InnerVolumeSpecName "kube-api-access-f9rps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:02.926585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.926555    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:35:02.959571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:02.959534    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9rps\" (UniqueName: \"kubernetes.io/projected/950eeb12-a96d-4fac-b7c2-091cd8fa93c2-kube-api-access-f9rps\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.060819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.060782    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-credential-socket\") pod \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") "
Oct 30 19:35:03.061011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.060834    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-certs\") pod \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") "
Oct 30 19:35:03.061011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.060897    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-podinfo\") pod \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") "
Oct 30 19:35:03.061152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061108    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-credential-socket" pod "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" (UID: "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.061193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061110    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-data\") pod \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") "
Oct 30 19:35:03.061193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061176    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-certs" pod "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" (UID: "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.061280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061207    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istiod-ca-cert\") pod \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") "
Oct 30 19:35:03.061280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061249    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-kube-api-access-vcxlr\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-kube-api-access-vcxlr\") pod \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") "
Oct 30 19:35:03.061390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061303    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-data" pod "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" (UID: "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.061390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061373    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-envoy\") pod \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") "
Oct 30 19:35:03.061511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061410    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-socket\") pod \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") "
Oct 30 19:35:03.061564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061510    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-token\") pod \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\" (UID: \"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c\") "
Oct 30 19:35:03.061617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061599    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istiod-ca-cert" pod "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" (UID: "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:03.061732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.061713    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-socket" pod "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" (UID: "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.062294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.062268    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.062294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.062293    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.062518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.062303    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.062518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.062312    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.062518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.062327    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.063923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.063884    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-podinfo" pod "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" (UID: "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:03.064555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.064526    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-envoy" pod "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" (UID: "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.064758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.064733    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-kube-api-access-vcxlr" pod "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" (UID: "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c"). InnerVolumeSpecName "kube-api-access-vcxlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:03.065371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.065343    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-token" pod "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" (UID: "d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:03.164033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.163993    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxlr\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-kube-api-access-vcxlr\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.164033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.164030    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.164033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.164041    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.164534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.164049    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.694640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.694608    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"
Oct 30 19:35:03.694867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.694638    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg" event={"ID":"950eeb12-a96d-4fac-b7c2-091cd8fa93c2","Type":"ContainerDied","Data":"41f72a751a2d7003b174be1bcfd89cd153c5d2fa2af9b8dc23da1df663f50ffc"}
Oct 30 19:35:03.694867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.694712    6777 scope.go:122] "RemoveContainer" containerID="f29a1b6b4dbf891d9a41fdbe9666fc66ae4964d84de5d91a4a335a7147f216aa"
Oct 30 19:35:03.696524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.696497    6777 generic.go:356] "Generic (PLEG): container finished" podID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerID="89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709" exitCode=0
Oct 30 19:35:03.696629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.696541    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"
Oct 30 19:35:03.696629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.696587    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" event={"ID":"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c","Type":"ContainerDied","Data":"89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709"}
Oct 30 19:35:03.696749 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.696639    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9" event={"ID":"d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c","Type":"ContainerDied","Data":"e400f237387537b9921eeeac6d1fc2b8bee844a8841f29c9b32c50601ff6c98c"}
Oct 30 19:35:03.707858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.707704    6777 scope.go:122] "RemoveContainer" containerID="89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709"
Oct 30 19:35:03.719530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.719513    6777 scope.go:122] "RemoveContainer" containerID="89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709"
Oct 30 19:35:03.719812 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:03.719792    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709\": container with ID starting with 89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709 not found: ID does not exist" containerID="89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709"
Oct 30 19:35:03.719866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.719820    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709"} err="failed to get container status \"89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709\": rpc error: code = NotFound desc = could not find container \"89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709\": container with ID starting with 89a66f9ea2ec3d5b83225fc4e26228ca577650aa6b3b25188bddba5a9617a709 not found: ID does not exist"
Oct 30 19:35:03.730627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.730444    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"]
Oct 30 19:35:03.738110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.738082    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-ltsg9"]
Oct 30 19:35:03.751661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.751622    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"]
Oct 30 19:35:03.755912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:03.755888    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afdf16-86c778d99f-xr7wg"]
Oct 30 19:35:04.115556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.115521    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950eeb12-a96d-4fac-b7c2-091cd8fa93c2" path="/var/lib/kubelet/pods/950eeb12-a96d-4fac-b7c2-091cd8fa93c2/volumes"
Oct 30 19:35:04.116249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.116226    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" path="/var/lib/kubelet/pods/d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c/volumes"
Oct 30 19:35:04.281197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.281159    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh" podUID="1d962588-5797-46dd-9413-e4c2efcf21f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:04.624271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.624229    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"]
Oct 30 19:35:04.625242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625221    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f07f718a-19bb-4e4c-a4c4-f6a10b687225" containerName="main"
Oct 30 19:35:04.625306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625246    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07f718a-19bb-4e4c-a4c4-f6a10b687225" containerName="main"
Oct 30 19:35:04.625306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625280    6777 cpu_manager.go:414] "containerMap: removing container" podUID="5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a" containerName="revoke-keys"
Oct 30 19:35:04.625306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625285    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a" containerName="revoke-keys"
Oct 30 19:35:04.625306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625301    6777 cpu_manager.go:414] "containerMap: removing container" podUID="950eeb12-a96d-4fac-b7c2-091cd8fa93c2" containerName="payload-pre-processing"
Oct 30 19:35:04.625306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625306    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950eeb12-a96d-4fac-b7c2-091cd8fa93c2" containerName="payload-pre-processing"
Oct 30 19:35:04.625503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625314    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy"
Oct 30 19:35:04.625503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625319    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy"
Oct 30 19:35:04.625503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625459    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a0608a-109b-4aef-88ab-52b53be4ca45" containerName="istio-proxy"
Oct 30 19:35:04.625503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625468    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="950eeb12-a96d-4fac-b7c2-091cd8fa93c2" containerName="payload-pre-processing"
Oct 30 19:35:04.625503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625474    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c6e626-bfc1-47cf-bb9a-5828cb65ac8c" containerName="istio-proxy"
Oct 30 19:35:04.625503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625490    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07f718a-19bb-4e4c-a4c4-f6a10b687225" containerName="main"
Oct 30 19:35:04.625503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.625501    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a" containerName="revoke-keys"
Oct 30 19:35:04.631143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.631119    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:04.645534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.645505    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ddb6ea\""
Oct 30 19:35:04.649510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.649482    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"]
Oct 30 19:35:04.654589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.654562    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ddb6ea-dockercfg-jnl5k\""
Oct 30 19:35:04.657700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.657672    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"]
Oct 30 19:35:04.664155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.664132    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:04.684658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:04.684621    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"]
Oct 30 19:35:05.063157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.023560    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e67fd308-ff22-476f-82a1-e22d41c8381e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp\" (UID: \"e67fd308-ff22-476f-82a1-e22d41c8381e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:05.064245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.063450    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c2e53-0ad0-40f6-af86-0bad69490983-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr\" (UID: \"095c2e53-0ad0-40f6-af86-0bad69490983\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:05.064245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.063561    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz45s\" (UniqueName: \"kubernetes.io/projected/e67fd308-ff22-476f-82a1-e22d41c8381e-kube-api-access-rz45s\") pod \"payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp\" (UID: \"e67fd308-ff22-476f-82a1-e22d41c8381e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:05.064245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.063601    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfn78\" (UniqueName: \"kubernetes.io/projected/095c2e53-0ad0-40f6-af86-0bad69490983-kube-api-access-cfn78\") pod \"payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr\" (UID: \"095c2e53-0ad0-40f6-af86-0bad69490983\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:05.166477 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.166431    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e67fd308-ff22-476f-82a1-e22d41c8381e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp\" (UID: \"e67fd308-ff22-476f-82a1-e22d41c8381e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:05.166661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.166546    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c2e53-0ad0-40f6-af86-0bad69490983-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr\" (UID: \"095c2e53-0ad0-40f6-af86-0bad69490983\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:05.166661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.166590    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz45s\" (UniqueName: \"kubernetes.io/projected/e67fd308-ff22-476f-82a1-e22d41c8381e-kube-api-access-rz45s\") pod \"payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp\" (UID: \"e67fd308-ff22-476f-82a1-e22d41c8381e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:05.166661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.166611    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfn78\" (UniqueName: \"kubernetes.io/projected/095c2e53-0ad0-40f6-af86-0bad69490983-kube-api-access-cfn78\") pod \"payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr\" (UID: \"095c2e53-0ad0-40f6-af86-0bad69490983\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:05.167366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.167341    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e67fd308-ff22-476f-82a1-e22d41c8381e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp\" (UID: \"e67fd308-ff22-476f-82a1-e22d41c8381e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:05.167549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.167519    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c2e53-0ad0-40f6-af86-0bad69490983-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr\" (UID: \"095c2e53-0ad0-40f6-af86-0bad69490983\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:05.185563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.185531    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfn78\" (UniqueName: \"kubernetes.io/projected/095c2e53-0ad0-40f6-af86-0bad69490983-kube-api-access-cfn78\") pod \"payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr\" (UID: \"095c2e53-0ad0-40f6-af86-0bad69490983\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:05.185563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.185551    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz45s\" (UniqueName: \"kubernetes.io/projected/e67fd308-ff22-476f-82a1-e22d41c8381e-kube-api-access-rz45s\") pod \"payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp\" (UID: \"e67fd308-ff22-476f-82a1-e22d41c8381e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:05.248622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.248591    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:05.284042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.283992    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:05.556205 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:35:05.556172    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095c2e53_0ad0_40f6_af86_0bad69490983.slice/crio-3eb77e35976bed2489a9f017075d6fbe4e32984be2557d989ac183c7f6b4320e WatchSource:0}: Error finding container 3eb77e35976bed2489a9f017075d6fbe4e32984be2557d989ac183c7f6b4320e: Status 404 returned error can't find the container with id 3eb77e35976bed2489a9f017075d6fbe4e32984be2557d989ac183c7f6b4320e
Oct 30 19:35:05.559032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.558998    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"]
Oct 30 19:35:05.598994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:05.598958    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"]
Oct 30 19:35:05.602809 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:35:05.602775    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67fd308_ff22_476f_82a1_e22d41c8381e.slice/crio-b7f21168abac1771f55d7a9b85d68b8c7372b2fd09329bda7a2c1f105288620b WatchSource:0}: Error finding container b7f21168abac1771f55d7a9b85d68b8c7372b2fd09329bda7a2c1f105288620b: Status 404 returned error can't find the container with id b7f21168abac1771f55d7a9b85d68b8c7372b2fd09329bda7a2c1f105288620b
Oct 30 19:35:06.101906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:06.101867    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp" event={"ID":"e67fd308-ff22-476f-82a1-e22d41c8381e","Type":"ContainerStarted","Data":"32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8"}
Oct 30 19:35:06.102256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:06.101926    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp" event={"ID":"e67fd308-ff22-476f-82a1-e22d41c8381e","Type":"ContainerStarted","Data":"b7f21168abac1771f55d7a9b85d68b8c7372b2fd09329bda7a2c1f105288620b"}
Oct 30 19:35:06.102256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:06.101955    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:06.104131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:06.104104    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr" event={"ID":"095c2e53-0ad0-40f6-af86-0bad69490983","Type":"ContainerStarted","Data":"976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58"}
Oct 30 19:35:06.104236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:06.104146    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr" event={"ID":"095c2e53-0ad0-40f6-af86-0bad69490983","Type":"ContainerStarted","Data":"3eb77e35976bed2489a9f017075d6fbe4e32984be2557d989ac183c7f6b4320e"}
Oct 30 19:35:06.104236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:06.104190    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:06.122176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:06.122125    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp" podStartSLOduration=2.122111445 podStartE2EDuration="2.122111445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:06.121050634 +0000 UTC m=+1688.117966023" watchObservedRunningTime="2026-10-30 19:35:06.122111445 +0000 UTC m=+1688.119026821"
Oct 30 19:35:06.153902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:06.153848    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr" podStartSLOduration=2.153833189 podStartE2EDuration="2.153833189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:06.151674191 +0000 UTC m=+1688.148589582" watchObservedRunningTime="2026-10-30 19:35:06.153833189 +0000 UTC m=+1688.150748590"
Oct 30 19:35:07.149085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:07.148229    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"]
Oct 30 19:35:07.195081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:07.182046    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d52364-rm7xg"]
Oct 30 19:35:08.128130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:08.128050    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a8e66c-42d3-44bf-b49b-2450713eb933" path="/var/lib/kubelet/pods/76a8e66c-42d3-44bf-b49b-2450713eb933/volumes"
Oct 30 19:35:10.719962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:10.719921    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j" podUID="7d475743-3ee9-452a-bd4c-6de427d0a778" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:13.642619 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:13.642578    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:35:14.279272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:14.279227    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh" podUID="1d962588-5797-46dd-9413-e4c2efcf21f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:16.678245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.678209    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"]
Oct 30 19:35:16.685877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.684830    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:16.691201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.691174    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"]
Oct 30 19:35:16.703859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.703822    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c86696-fblsj\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:16.704035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.704013    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx877\" (UniqueName: \"kubernetes.io/projected/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-kube-api-access-dx877\") pod \"maas-api-revoke-keys-e2e-inf-b-c86696-fblsj\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:16.704205 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.704184    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c86696-fblsj\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:16.757756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.757726    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:16.757756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.757765    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:35:16.805934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.805890    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx877\" (UniqueName: \"kubernetes.io/projected/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-kube-api-access-dx877\") pod \"maas-api-revoke-keys-e2e-inf-b-c86696-fblsj\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:16.806183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.806026    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c86696-fblsj\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:16.806271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.806255    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c86696-fblsj\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:16.806895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.806869    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c86696-fblsj\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:16.808569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.808549    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c86696-fblsj\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:16.834034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:16.834004    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx877\" (UniqueName: \"kubernetes.io/projected/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-kube-api-access-dx877\") pod \"maas-api-revoke-keys-e2e-inf-b-c86696-fblsj\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:17.006857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:17.006778    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:17.187855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:17.187810    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:35:17.190221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:17.190190    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:35:17.308566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:17.308533    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"]
Oct 30 19:35:17.494551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:17.494514    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj" event={"ID":"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3","Type":"ContainerStarted","Data":"ae72a51d02a1e9516691012496a534d3f1642ab0d34451d0d843abff5c2033ed"}
Oct 30 19:35:18.508346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:18.508301    6777 generic.go:356] "Generic (PLEG): container finished" podID="ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3" containerID="eac0b33024879d42f748ae091d8a5e7231017dc3a1c3ba1484447285fd86b0c6" exitCode=0
Oct 30 19:35:18.508822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:18.508386    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj" event={"ID":"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3","Type":"ContainerDied","Data":"eac0b33024879d42f748ae091d8a5e7231017dc3a1c3ba1484447285fd86b0c6"}
Oct 30 19:35:19.717133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.717093    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"]
Oct 30 19:35:19.770375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.770298    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:19.875153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.875110    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"]
Oct 30 19:35:19.875413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.875385    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" containerID="cri-o://b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a" gracePeriod=30
Oct 30 19:35:19.880045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.880014    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-xks-ca\") pod \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") "
Oct 30 19:35:19.880216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.880193    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-openshift-ca\") pod \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") "
Oct 30 19:35:19.880351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.880249    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-kube-api-access-dx877\" (UniqueName: \"kubernetes.io/projected/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-kube-api-access-dx877\") pod \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\" (UID: \"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3\") "
Oct 30 19:35:19.880655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.880628    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-openshift-ca" pod "ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3" (UID: "ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:19.881287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.881261    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.881825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.881801    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"]
Oct 30 19:35:19.882152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.882118    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" containerID="cri-o://7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5" gracePeriod=30
Oct 30 19:35:19.883575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.883547    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-xks-ca" pod "ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3" (UID: "ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:19.883686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.883635    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-kube-api-access-dx877" pod "ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3" (UID: "ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3"). InnerVolumeSpecName "kube-api-access-dx877". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:19.911720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.910365    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:35:19.911720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.910511    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:35:19.983413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.983376    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.983413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:19.983407    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx877\" (UniqueName: \"kubernetes.io/projected/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3-kube-api-access-dx877\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:20.042640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.042604    6777 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:35:20.042811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.042677    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:35:20.447401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.447227    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"]
Oct 30 19:35:20.448153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.448129    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3" containerName="revoke-keys"
Oct 30 19:35:20.448153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.448148    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3" containerName="revoke-keys"
Oct 30 19:35:20.448340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.448297    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3" containerName="revoke-keys"
Oct 30 19:35:20.453094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.452726    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.459146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.459121    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"]
Oct 30 19:35:20.501941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.501863    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.502129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.502039    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bchr7\" (UniqueName: \"kubernetes.io/projected/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-kube-api-access-bchr7\") pod \"maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.502213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.502136    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.531246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.531218    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"
Oct 30 19:35:20.531481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.531272    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj" event={"ID":"ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3","Type":"ContainerDied","Data":"ae72a51d02a1e9516691012496a534d3f1642ab0d34451d0d843abff5c2033ed"}
Oct 30 19:35:20.531584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.531516    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae72a51d02a1e9516691012496a534d3f1642ab0d34451d0d843abff5c2033ed"
Oct 30 19:35:20.603822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.603782    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.604028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.603851    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bchr7\" (UniqueName: \"kubernetes.io/projected/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-kube-api-access-bchr7\") pod \"maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.604028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.603880    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.604573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.604545    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.606567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.606547    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.640051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.640022    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bchr7\" (UniqueName: \"kubernetes.io/projected/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-kube-api-access-bchr7\") pod \"maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.694322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.694283    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:35:20.705759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.705730    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d475743-3ee9-452a-bd4c-6de427d0a778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d475743-3ee9-452a-bd4c-6de427d0a778-plugins-config-volume\") pod \"7d475743-3ee9-452a-bd4c-6de427d0a778\" (UID: \"7d475743-3ee9-452a-bd4c-6de427d0a778\") "
Oct 30 19:35:20.705994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.705979    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d475743-3ee9-452a-bd4c-6de427d0a778-kube-api-access-zr2hc\" (UniqueName: \"kubernetes.io/projected/7d475743-3ee9-452a-bd4c-6de427d0a778-kube-api-access-zr2hc\") pod \"7d475743-3ee9-452a-bd4c-6de427d0a778\" (UID: \"7d475743-3ee9-452a-bd4c-6de427d0a778\") "
Oct 30 19:35:20.706199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.706172    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d475743-3ee9-452a-bd4c-6de427d0a778-plugins-config-volume" pod "7d475743-3ee9-452a-bd4c-6de427d0a778" (UID: "7d475743-3ee9-452a-bd4c-6de427d0a778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:20.707045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.707009    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d475743-3ee9-452a-bd4c-6de427d0a778-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:20.708771 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.708738    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d475743-3ee9-452a-bd4c-6de427d0a778-kube-api-access-zr2hc" pod "7d475743-3ee9-452a-bd4c-6de427d0a778" (UID: "7d475743-3ee9-452a-bd4c-6de427d0a778"). InnerVolumeSpecName "kube-api-access-zr2hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:20.806456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.806421    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:20.808654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:20.808624    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2hc\" (UniqueName: \"kubernetes.io/projected/7d475743-3ee9-452a-bd4c-6de427d0a778-kube-api-access-zr2hc\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.114268 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:35:21.114232    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8316b0da_7ad2_4fe5_8cef_b47a2289f3e4.slice/crio-dc189d4f97ad8fc72b3ce379e191a082d08977a5f35234c04c1ba8fdefa2e16d WatchSource:0}: Error finding container dc189d4f97ad8fc72b3ce379e191a082d08977a5f35234c04c1ba8fdefa2e16d: Status 404 returned error can't find the container with id dc189d4f97ad8fc72b3ce379e191a082d08977a5f35234c04c1ba8fdefa2e16d
Oct 30 19:35:21.114932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.114901    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"]
Oct 30 19:35:21.381565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.381538    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:35:21.422388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.422351    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d962588-5797-46dd-9413-e4c2efcf21f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d962588-5797-46dd-9413-e4c2efcf21f8-plugins-config-volume\") pod \"1d962588-5797-46dd-9413-e4c2efcf21f8\" (UID: \"1d962588-5797-46dd-9413-e4c2efcf21f8\") "
Oct 30 19:35:21.422602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.422582    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d962588-5797-46dd-9413-e4c2efcf21f8-kube-api-access-nsj9t\" (UniqueName: \"kubernetes.io/projected/1d962588-5797-46dd-9413-e4c2efcf21f8-kube-api-access-nsj9t\") pod \"1d962588-5797-46dd-9413-e4c2efcf21f8\" (UID: \"1d962588-5797-46dd-9413-e4c2efcf21f8\") "
Oct 30 19:35:21.422885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.422855    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d962588-5797-46dd-9413-e4c2efcf21f8-plugins-config-volume" pod "1d962588-5797-46dd-9413-e4c2efcf21f8" (UID: "1d962588-5797-46dd-9413-e4c2efcf21f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:21.423876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.423845    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d962588-5797-46dd-9413-e4c2efcf21f8-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.425150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.425122    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d962588-5797-46dd-9413-e4c2efcf21f8-kube-api-access-nsj9t" pod "1d962588-5797-46dd-9413-e4c2efcf21f8" (UID: "1d962588-5797-46dd-9413-e4c2efcf21f8"). InnerVolumeSpecName "kube-api-access-nsj9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:21.525727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.525691    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsj9t\" (UniqueName: \"kubernetes.io/projected/1d962588-5797-46dd-9413-e4c2efcf21f8-kube-api-access-nsj9t\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.543027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.542993    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv" event={"ID":"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4","Type":"ContainerStarted","Data":"dc189d4f97ad8fc72b3ce379e191a082d08977a5f35234c04c1ba8fdefa2e16d"}
Oct 30 19:35:21.544856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.544829    6777 generic.go:356] "Generic (PLEG): container finished" podID="1d962588-5797-46dd-9413-e4c2efcf21f8" containerID="1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479" exitCode=0
Oct 30 19:35:21.544979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.544877    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"
Oct 30 19:35:21.544979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.544918    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh" event={"ID":"1d962588-5797-46dd-9413-e4c2efcf21f8","Type":"ContainerDied","Data":"1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479"}
Oct 30 19:35:21.545114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.544984    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh" event={"ID":"1d962588-5797-46dd-9413-e4c2efcf21f8","Type":"ContainerDied","Data":"4d99124b9ff362f51f87ce450d4eaa8e86b5402a5abadca245a2e5f82cef44b7"}
Oct 30 19:35:21.545114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.545016    6777 scope.go:122] "RemoveContainer" containerID="1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479"
Oct 30 19:35:21.546717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.546695    6777 generic.go:356] "Generic (PLEG): container finished" podID="7d475743-3ee9-452a-bd4c-6de427d0a778" containerID="c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747" exitCode=0
Oct 30 19:35:21.546836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.546721    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j" event={"ID":"7d475743-3ee9-452a-bd4c-6de427d0a778","Type":"ContainerDied","Data":"c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747"}
Oct 30 19:35:21.546836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.546761    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"
Oct 30 19:35:21.546836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.546781    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j" event={"ID":"7d475743-3ee9-452a-bd4c-6de427d0a778","Type":"ContainerDied","Data":"dc76ffe9940f5fdc9236c2420e179be30850e20d30417600fb50874237234e91"}
Oct 30 19:35:21.559627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.559603    6777 scope.go:122] "RemoveContainer" containerID="1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479"
Oct 30 19:35:21.560194 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:21.560142    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479\": container with ID starting with 1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479 not found: ID does not exist" containerID="1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479"
Oct 30 19:35:21.560194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.560180    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479"} err="failed to get container status \"1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479\": rpc error: code = NotFound desc = could not find container \"1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479\": container with ID starting with 1f95fc7b2539c857ab11c59e9f44fa56c38c870988f71abf8be434c3c1ecb479 not found: ID does not exist"
Oct 30 19:35:21.560412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.560206    6777 scope.go:122] "RemoveContainer" containerID="c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747"
Oct 30 19:35:21.573141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.573051    6777 scope.go:122] "RemoveContainer" containerID="c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747"
Oct 30 19:35:21.573465 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:21.573444    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747\": container with ID starting with c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747 not found: ID does not exist" containerID="c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747"
Oct 30 19:35:21.573526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.573473    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747"} err="failed to get container status \"c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747\": rpc error: code = NotFound desc = could not find container \"c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747\": container with ID starting with c9dd49ac74b9a1009885fb1119c8b0bbf9ebc166e1aaff0371098d4322004747 not found: ID does not exist"
Oct 30 19:35:21.581894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.581868    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"]
Oct 30 19:35:21.586415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.586389    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-02feaa-77cc96dff5-nn57j"]
Oct 30 19:35:21.603172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.603150    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"]
Oct 30 19:35:21.622733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:21.622648    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d52364-8544558f6c-v5fvh"]
Oct 30 19:35:22.152543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:22.152499    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d962588-5797-46dd-9413-e4c2efcf21f8" path="/var/lib/kubelet/pods/1d962588-5797-46dd-9413-e4c2efcf21f8/volumes"
Oct 30 19:35:22.153452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:22.153424    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d475743-3ee9-452a-bd4c-6de427d0a778" path="/var/lib/kubelet/pods/7d475743-3ee9-452a-bd4c-6de427d0a778/volumes"
Oct 30 19:35:22.559498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:22.559457    6777 generic.go:356] "Generic (PLEG): container finished" podID="8316b0da-7ad2-4fe5-8cef-b47a2289f3e4" containerID="c3edf7eaa94c3100f3ad4fa7de8124da2d09de7031e7c8574c70e38517b2d6ca" exitCode=0
Oct 30 19:35:22.559687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:22.559544    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv" event={"ID":"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4","Type":"ContainerDied","Data":"c3edf7eaa94c3100f3ad4fa7de8124da2d09de7031e7c8574c70e38517b2d6ca"}
Oct 30 19:35:23.778372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.778345    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:23.886151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.886031    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-xks-ca\") pod \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") "
Oct 30 19:35:23.886151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.886144    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-kube-api-access-bchr7\" (UniqueName: \"kubernetes.io/projected/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-kube-api-access-bchr7\") pod \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") "
Oct 30 19:35:23.886402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.886355    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-openshift-ca\") pod \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\" (UID: \"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4\") "
Oct 30 19:35:23.886707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.886679    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-openshift-ca" pod "8316b0da-7ad2-4fe5-8cef-b47a2289f3e4" (UID: "8316b0da-7ad2-4fe5-8cef-b47a2289f3e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:23.887096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.887048    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:23.888234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.888208    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-xks-ca" pod "8316b0da-7ad2-4fe5-8cef-b47a2289f3e4" (UID: "8316b0da-7ad2-4fe5-8cef-b47a2289f3e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:23.888311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.888293    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-kube-api-access-bchr7" pod "8316b0da-7ad2-4fe5-8cef-b47a2289f3e4" (UID: "8316b0da-7ad2-4fe5-8cef-b47a2289f3e4"). InnerVolumeSpecName "kube-api-access-bchr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:23.988970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.988923    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:23.988970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:23.988961    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bchr7\" (UniqueName: \"kubernetes.io/projected/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4-kube-api-access-bchr7\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:24.582663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:24.582635    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"
Oct 30 19:35:24.582847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:24.582678    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv" event={"ID":"8316b0da-7ad2-4fe5-8cef-b47a2289f3e4","Type":"ContainerDied","Data":"dc189d4f97ad8fc72b3ce379e191a082d08977a5f35234c04c1ba8fdefa2e16d"}
Oct 30 19:35:24.582847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:24.582732    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc189d4f97ad8fc72b3ce379e191a082d08977a5f35234c04c1ba8fdefa2e16d"
Oct 30 19:35:25.256512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.256491    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:35:25.306294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.305633    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:35:25.327941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.326624    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-socket\") pod \"f3692e68-9f66-4f51-b682-3d91a22b187e\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") "
Oct 30 19:35:25.327941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.327041    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-envoy\") pod \"f3692e68-9f66-4f51-b682-3d91a22b187e\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") "
Oct 30 19:35:25.327941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.327178    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3692e68-9f66-4f51-b682-3d91a22b187e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3692e68-9f66-4f51-b682-3d91a22b187e-istiod-ca-cert\") pod \"f3692e68-9f66-4f51-b682-3d91a22b187e\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") "
Oct 30 19:35:25.327941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.327219    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-data\") pod \"f3692e68-9f66-4f51-b682-3d91a22b187e\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") "
Oct 30 19:35:25.327941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.327285    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-certs\") pod \"f3692e68-9f66-4f51-b682-3d91a22b187e\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") "
Oct 30 19:35:25.327941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.327429    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-token\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-token\") pod \"f3692e68-9f66-4f51-b682-3d91a22b187e\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") "
Oct 30 19:35:25.327941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.327481    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-kube-api-access-gx2q4\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-kube-api-access-gx2q4\") pod \"f3692e68-9f66-4f51-b682-3d91a22b187e\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") "
Oct 30 19:35:25.327941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.327603    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-podinfo\") pod \"f3692e68-9f66-4f51-b682-3d91a22b187e\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") "
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.326930    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-socket" pod "f3692e68-9f66-4f51-b682-3d91a22b187e" (UID: "f3692e68-9f66-4f51-b682-3d91a22b187e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.328740    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-data" pod "f3692e68-9f66-4f51-b682-3d91a22b187e" (UID: "f3692e68-9f66-4f51-b682-3d91a22b187e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.330326    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-certs" pod "f3692e68-9f66-4f51-b682-3d91a22b187e" (UID: "f3692e68-9f66-4f51-b682-3d91a22b187e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.330155    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-credential-socket" pod "f3692e68-9f66-4f51-b682-3d91a22b187e" (UID: "f3692e68-9f66-4f51-b682-3d91a22b187e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.331359    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3692e68-9f66-4f51-b682-3d91a22b187e-istiod-ca-cert" pod "f3692e68-9f66-4f51-b682-3d91a22b187e" (UID: "f3692e68-9f66-4f51-b682-3d91a22b187e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.332885    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-podinfo" pod "f3692e68-9f66-4f51-b682-3d91a22b187e" (UID: "f3692e68-9f66-4f51-b682-3d91a22b187e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.333949    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-token" pod "f3692e68-9f66-4f51-b682-3d91a22b187e" (UID: "f3692e68-9f66-4f51-b682-3d91a22b187e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.334042    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-envoy" pod "f3692e68-9f66-4f51-b682-3d91a22b187e" (UID: "f3692e68-9f66-4f51-b682-3d91a22b187e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.334337    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-kube-api-access-gx2q4" pod "f3692e68-9f66-4f51-b682-3d91a22b187e" (UID: "f3692e68-9f66-4f51-b682-3d91a22b187e"). InnerVolumeSpecName "kube-api-access-gx2q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.329866    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-credential-socket\") pod \"f3692e68-9f66-4f51-b682-3d91a22b187e\" (UID: \"f3692e68-9f66-4f51-b682-3d91a22b187e\") "
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.335308    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx2q4\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-kube-api-access-gx2q4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.335331    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.335347    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.335363    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.335377    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.335390    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3692e68-9f66-4f51-b682-3d91a22b187e-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.335404    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.335416    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3692e68-9f66-4f51-b682-3d91a22b187e-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.336452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.335432    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3692e68-9f66-4f51-b682-3d91a22b187e-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.439432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.437638    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-envoy\") pod \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") "
Oct 30 19:35:25.439794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.439774    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-credential-socket\") pod \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") "
Oct 30 19:35:25.440261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.440241    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-token\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-token\") pod \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") "
Oct 30 19:35:25.440801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.440783    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-certs\") pod \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") "
Oct 30 19:35:25.441251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.441235    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istiod-ca-cert\") pod \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") "
Oct 30 19:35:25.441695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.441678    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-data\") pod \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") "
Oct 30 19:35:25.442150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.442132    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-podinfo\") pod \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") "
Oct 30 19:35:25.442635 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.440165    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-credential-socket" pod "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" (UID: "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.442796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.441153    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-certs" pod "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" (UID: "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.442796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.441589    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istiod-ca-cert" pod "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" (UID: "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:25.442796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.442004    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-data" pod "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" (UID: "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.443483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.443463    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-kube-api-access-24xgt\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-kube-api-access-24xgt\") pod \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") "
Oct 30 19:35:25.444122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.444083    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-socket" pod "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" (UID: "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.444398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.443667    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-socket\") pod \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\" (UID: \"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799\") "
Oct 30 19:35:25.446693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.446674    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.446835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.446822    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.446925 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.446912    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.447110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.447093    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.447221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.447209    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.511923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.511857    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-kube-api-access-24xgt" pod "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" (UID: "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799"). InnerVolumeSpecName "kube-api-access-24xgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:25.523396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.523341    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-podinfo" pod "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" (UID: "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:25.542089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.538654    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-envoy" pod "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" (UID: "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.542629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.542569    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-token" pod "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" (UID: "d61c55b9-72e4-47a6-b8df-1f8ac5ecf799"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:25.595170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.580783    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24xgt\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-kube-api-access-24xgt\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.595170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.580821    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.595170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.580837    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.595170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.580852    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.665540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.665497    6777 generic.go:356] "Generic (PLEG): container finished" podID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerID="7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5" exitCode=0
Oct 30 19:35:25.665749 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.665701    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"
Oct 30 19:35:25.665830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.665783    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" event={"ID":"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799","Type":"ContainerDied","Data":"7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5"}
Oct 30 19:35:25.665884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.665870    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj" event={"ID":"d61c55b9-72e4-47a6-b8df-1f8ac5ecf799","Type":"ContainerDied","Data":"acc2a0c8d9db1842d645cc0fcc934b8215f6082f9a605f3ad38a24a5b5bbf84d"}
Oct 30 19:35:25.665943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.665909    6777 scope.go:122] "RemoveContainer" containerID="7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5"
Oct 30 19:35:25.671562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.671518    6777 generic.go:356] "Generic (PLEG): container finished" podID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerID="b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a" exitCode=0
Oct 30 19:35:25.673883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.671898    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"
Oct 30 19:35:25.674972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.671918    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" event={"ID":"f3692e68-9f66-4f51-b682-3d91a22b187e","Type":"ContainerDied","Data":"b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a"}
Oct 30 19:35:25.678572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.678546    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2" event={"ID":"f3692e68-9f66-4f51-b682-3d91a22b187e","Type":"ContainerDied","Data":"322e5853fd6c17ded07a00f4e22690d01378e4a9618bbda0eb821e7e2b9444ad"}
Oct 30 19:35:25.710838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.710815    6777 scope.go:122] "RemoveContainer" containerID="14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3"
Oct 30 19:35:25.725504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.725477    6777 scope.go:122] "RemoveContainer" containerID="7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5"
Oct 30 19:35:25.725978 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:25.725948    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5\": container with ID starting with 7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5 not found: ID does not exist" containerID="7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5"
Oct 30 19:35:25.726144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.725987    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5"} err="failed to get container status \"7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5\": rpc error: code = NotFound desc = could not find container \"7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5\": container with ID starting with 7d605380bf5b9a458bbc294c8b678874bedb7b20f220e53a23fcb8b4af2557d5 not found: ID does not exist"
Oct 30 19:35:25.726144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.726011    6777 scope.go:122] "RemoveContainer" containerID="14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3"
Oct 30 19:35:25.726570 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:25.726538    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3\": container with ID starting with 14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3 not found: ID does not exist" containerID="14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3"
Oct 30 19:35:25.726638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.726582    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3"} err="failed to get container status \"14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3\": rpc error: code = NotFound desc = could not find container \"14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3\": container with ID starting with 14002c3cbfefc1c6a306270ac3e1bc0f773fcb2c5b29f68b63bd9498313b4bc3 not found: ID does not exist"
Oct 30 19:35:25.726638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.726607    6777 scope.go:122] "RemoveContainer" containerID="b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a"
Oct 30 19:35:25.751188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.751162    6777 scope.go:122] "RemoveContainer" containerID="b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a"
Oct 30 19:35:25.751754 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:25.751727    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a\": container with ID starting with b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a not found: ID does not exist" containerID="b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a"
Oct 30 19:35:25.751871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.751763    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a"} err="failed to get container status \"b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a\": rpc error: code = NotFound desc = could not find container \"b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a\": container with ID starting with b77d048edab779821e78ecb09360bd032ee64a7f98d2aff14daadec4b29f2f7a not found: ID does not exist"
Oct 30 19:35:25.760998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.760930    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"]
Oct 30 19:35:25.766130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.766086    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-xf8vj"]
Oct 30 19:35:25.780329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.780299    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"]
Oct 30 19:35:25.789641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:25.789607    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-th9n2"]
Oct 30 19:35:26.115509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:26.115477    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" path="/var/lib/kubelet/pods/d61c55b9-72e4-47a6-b8df-1f8ac5ecf799/volumes"
Oct 30 19:35:26.116141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:26.116125    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" path="/var/lib/kubelet/pods/f3692e68-9f66-4f51-b682-3d91a22b187e/volumes"
Oct 30 19:35:26.737110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:26.737072    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"]
Oct 30 19:35:26.754004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:26.753969    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"]
Oct 30 19:35:31.213664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:31.212582    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"]
Oct 30 19:35:31.237075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:31.234376    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"]
Oct 30 19:35:31.436892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:31.436849    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:35:31.739079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:31.738398    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8" podUID="a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" containerName="payload-processing" containerID="cri-o://cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954" gracePeriod=25
Oct 30 19:35:31.755084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:31.754638    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8" podUID="eaf2532e-7980-44d9-b14e-650e57d53c1c" containerName="payload-pre-processing" containerID="cri-o://b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052" gracePeriod=25
Oct 30 19:35:34.801964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:34.797398    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5" podUID="0da1619c-e802-47a9-abff-7406f07cf890" containerName="main" containerID="cri-o://74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c" gracePeriod=45
Oct 30 19:35:35.101793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.101710    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"]
Oct 30 19:35:35.106338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.106310    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c86696-fblsj"]
Oct 30 19:35:35.381288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.381211    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:35:35.422436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.422395    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-dshm\") pod \"0da1619c-e802-47a9-abff-7406f07cf890\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") "
Oct 30 19:35:35.422631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.422465    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da1619c-e802-47a9-abff-7406f07cf890-kube-api-access-mvk6q\" (UniqueName: \"kubernetes.io/projected/0da1619c-e802-47a9-abff-7406f07cf890-kube-api-access-mvk6q\") pod \"0da1619c-e802-47a9-abff-7406f07cf890\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") "
Oct 30 19:35:35.422706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.422641    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-home\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-home\") pod \"0da1619c-e802-47a9-abff-7406f07cf890\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") "
Oct 30 19:35:35.422805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.422777    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-tmp-dir\") pod \"0da1619c-e802-47a9-abff-7406f07cf890\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") "
Oct 30 19:35:35.422921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.422882    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-model-cache\") pod \"0da1619c-e802-47a9-abff-7406f07cf890\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") "
Oct 30 19:35:35.423010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.422913    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-home" pod "0da1619c-e802-47a9-abff-7406f07cf890" (UID: "0da1619c-e802-47a9-abff-7406f07cf890"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:35.423079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.423036    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da1619c-e802-47a9-abff-7406f07cf890-tls-certs\" (UniqueName: \"kubernetes.io/secret/0da1619c-e802-47a9-abff-7406f07cf890-tls-certs\") pod \"0da1619c-e802-47a9-abff-7406f07cf890\" (UID: \"0da1619c-e802-47a9-abff-7406f07cf890\") "
Oct 30 19:35:35.423130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.423046    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-tmp-dir" pod "0da1619c-e802-47a9-abff-7406f07cf890" (UID: "0da1619c-e802-47a9-abff-7406f07cf890"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:35.423243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.423219    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-model-cache" pod "0da1619c-e802-47a9-abff-7406f07cf890" (UID: "0da1619c-e802-47a9-abff-7406f07cf890"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:35.423973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.423944    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:35.424132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.423979    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:35.424132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.423994    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:35.425271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.425238    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da1619c-e802-47a9-abff-7406f07cf890-kube-api-access-mvk6q" pod "0da1619c-e802-47a9-abff-7406f07cf890" (UID: "0da1619c-e802-47a9-abff-7406f07cf890"). InnerVolumeSpecName "kube-api-access-mvk6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:35.425385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.425292    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-dshm" pod "0da1619c-e802-47a9-abff-7406f07cf890" (UID: "0da1619c-e802-47a9-abff-7406f07cf890"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:35.426170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.426146    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1619c-e802-47a9-abff-7406f07cf890-tls-certs" pod "0da1619c-e802-47a9-abff-7406f07cf890" (UID: "0da1619c-e802-47a9-abff-7406f07cf890"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:35.525770 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.525731    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da1619c-e802-47a9-abff-7406f07cf890-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:35.525770 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.525764    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da1619c-e802-47a9-abff-7406f07cf890-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:35.525770 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.525775    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvk6q\" (UniqueName: \"kubernetes.io/projected/0da1619c-e802-47a9-abff-7406f07cf890-kube-api-access-mvk6q\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:35.989189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.989154    6777 generic.go:356] "Generic (PLEG): container finished" podID="0da1619c-e802-47a9-abff-7406f07cf890" containerID="74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c" exitCode=0
Oct 30 19:35:35.989680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.989203    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"
Oct 30 19:35:35.989680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.989230    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5" event={"ID":"0da1619c-e802-47a9-abff-7406f07cf890","Type":"ContainerDied","Data":"74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c"}
Oct 30 19:35:35.989680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.989274    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5" event={"ID":"0da1619c-e802-47a9-abff-7406f07cf890","Type":"ContainerDied","Data":"64ee45e22a05447fdad02d896f9e88a3c264ac5a74036f856b36b4f3986b0635"}
Oct 30 19:35:35.989680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:35.989288    6777 scope.go:122] "RemoveContainer" containerID="74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c"
Oct 30 19:35:36.016691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.016657    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"]
Oct 30 19:35:36.020810 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.020778    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5"]
Oct 30 19:35:36.072420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.072387    6777 scope.go:122] "RemoveContainer" containerID="74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c"
Oct 30 19:35:36.073348 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:36.073316    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c\": container with ID starting with 74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c not found: ID does not exist" containerID="74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c"
Oct 30 19:35:36.073575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.073542    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c"} err="failed to get container status \"74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c\": rpc error: code = NotFound desc = could not find container \"74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c\": container with ID starting with 74b8a226ef670ccf09191f955e05b3af79fea0c06366745b3f098d7f62034c4c not found: ID does not exist"
Oct 30 19:35:36.126112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.123618    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da1619c-e802-47a9-abff-7406f07cf890" path="/var/lib/kubelet/pods/0da1619c-e802-47a9-abff-7406f07cf890/volumes"
Oct 30 19:35:36.126112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.124246    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3" path="/var/lib/kubelet/pods/ce8b9a2f-b3d2-4f62-a5d6-d3adbb2847f3/volumes"
Oct 30 19:35:36.212928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.212874    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54" podUID="dfc9bd5f-9cac-4cdb-861a-006ec25036fd" containerName="payload-processing" containerID="cri-o://847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd" gracePeriod=25
Oct 30 19:35:36.218371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.218342    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8" podUID="a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:36.218522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.218496    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8" podUID="eaf2532e-7980-44d9-b14e-650e57d53c1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:36.232787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.232746    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/auth-test-model-e3586f-kserve-84fdf76b5-7rsx5" podUID="0da1619c-e802-47a9-abff-7406f07cf890" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:35:36.234916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.234866    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn" podUID="2176303e-2380-47ad-81c5-afba1009961d" containerName="payload-pre-processing" containerID="cri-o://2c4be0ea519487ec959908a4e0f3259c59525ca4dae260e248b53e01de0f5217" gracePeriod=25
Oct 30 19:35:36.577105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.576282    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:35:36.644856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.644766    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-kube-api-access-2prdk\" (UniqueName: \"kubernetes.io/projected/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-kube-api-access-2prdk\") pod \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\" (UID: \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\") "
Oct 30 19:35:36.644998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.644941    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-plugins-config-volume\") pod \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\" (UID: \"dfc9bd5f-9cac-4cdb-861a-006ec25036fd\") "
Oct 30 19:35:36.645410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.645385    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-plugins-config-volume" pod "dfc9bd5f-9cac-4cdb-861a-006ec25036fd" (UID: "dfc9bd5f-9cac-4cdb-861a-006ec25036fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:36.645735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.645720    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:36.647043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.647017    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-kube-api-access-2prdk" pod "dfc9bd5f-9cac-4cdb-861a-006ec25036fd" (UID: "dfc9bd5f-9cac-4cdb-861a-006ec25036fd"). InnerVolumeSpecName "kube-api-access-2prdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:36.694865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.693324    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"]
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.697893    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7d475743-3ee9-452a-bd4c-6de427d0a778" containerName="payload-pre-processing"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.697921    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d475743-3ee9-452a-bd4c-6de427d0a778" containerName="payload-pre-processing"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.697939    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.697947    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.697971    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8316b0da-7ad2-4fe5-8cef-b47a2289f3e4" containerName="revoke-keys"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.697978    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8316b0da-7ad2-4fe5-8cef-b47a2289f3e4" containerName="revoke-keys"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.697996    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1d962588-5797-46dd-9413-e4c2efcf21f8" containerName="payload-pre-processing"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698005    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d962588-5797-46dd-9413-e4c2efcf21f8" containerName="payload-pre-processing"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698039    6777 cpu_manager.go:414] "containerMap: removing container" podUID="dfc9bd5f-9cac-4cdb-861a-006ec25036fd" containerName="payload-processing"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698047    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc9bd5f-9cac-4cdb-861a-006ec25036fd" containerName="payload-processing"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698089    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698099    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698136    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0da1619c-e802-47a9-abff-7406f07cf890" containerName="main"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698145    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da1619c-e802-47a9-abff-7406f07cf890" containerName="main"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698361    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da1619c-e802-47a9-abff-7406f07cf890" containerName="main"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698380    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d475743-3ee9-452a-bd4c-6de427d0a778" containerName="payload-pre-processing"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698391    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698404    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698413    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8316b0da-7ad2-4fe5-8cef-b47a2289f3e4" containerName="revoke-keys"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698423    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d962588-5797-46dd-9413-e4c2efcf21f8" containerName="payload-pre-processing"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698435    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc9bd5f-9cac-4cdb-861a-006ec25036fd" containerName="payload-processing"
Oct 30 19:35:36.699389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.698448    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3692e68-9f66-4f51-b682-3d91a22b187e" containerName="istio-proxy"
Oct 30 19:35:36.708234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.708207    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"]
Oct 30 19:35:36.709293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.708589    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.715602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.715501    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f2dbd9\"/\"openshift-service-ca.crt\""
Oct 30 19:35:36.715762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.715620    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f2dbd9\"/\"kube-root-ca.crt\""
Oct 30 19:35:36.738356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.738013    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f2dbd9\"/\"default-dockercfg-9l6qm\""
Oct 30 19:35:36.739192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.739130    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f2dbd9\"/\"test-model-f2dbd9-kserve-self-signed-certs\""
Oct 30 19:35:36.746951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.746920    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-dshm\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.747152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.747036    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-home\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.747228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.747176    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tmp-dir\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.747300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.747246    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcndl\" (UniqueName: \"kubernetes.io/projected/aff4e94e-4e6a-463e-84a4-60ae69538dd9-kube-api-access-qcndl\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.747300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.747277    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tls-certs\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.747420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.747329    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-model-cache\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.747480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.747452    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2prdk\" (UniqueName: \"kubernetes.io/projected/dfc9bd5f-9cac-4cdb-861a-006ec25036fd-kube-api-access-2prdk\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:36.848901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.848848    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-home\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.853147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.849045    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tmp-dir\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.853147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.849184    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcndl\" (UniqueName: \"kubernetes.io/projected/aff4e94e-4e6a-463e-84a4-60ae69538dd9-kube-api-access-qcndl\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.853147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.849229    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tls-certs\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.853147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.849298    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-model-cache\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.853147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.849427    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-dshm\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.876758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.863959    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tmp-dir\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.876758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.867503    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-model-cache\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.876758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.872302    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-home\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.919209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.915240    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcndl\" (UniqueName: \"kubernetes.io/projected/aff4e94e-4e6a-463e-84a4-60ae69538dd9-kube-api-access-qcndl\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:36.983082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:36.964490    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tls-certs\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:37.014381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.014343    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-dshm\") pod \"test-model-f2dbd9-kserve-5876c4484f-btw79\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") " pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:37.033397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.033359    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:37.124570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.124528    6777 generic.go:356] "Generic (PLEG): container finished" podID="dfc9bd5f-9cac-4cdb-861a-006ec25036fd" containerID="847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd" exitCode=0
Oct 30 19:35:37.125085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.125035    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54" event={"ID":"dfc9bd5f-9cac-4cdb-861a-006ec25036fd","Type":"ContainerDied","Data":"847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd"}
Oct 30 19:35:37.126380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.126355    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54" event={"ID":"dfc9bd5f-9cac-4cdb-861a-006ec25036fd","Type":"ContainerDied","Data":"ebc1116369d42381bb38ac0747787287cf53f5e16eb9bbced9e88b522f22429c"}
Oct 30 19:35:37.127241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.125876    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"
Oct 30 19:35:37.128786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.127006    6777 scope.go:122] "RemoveContainer" containerID="847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd"
Oct 30 19:35:37.225137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.225115    6777 scope.go:122] "RemoveContainer" containerID="847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd"
Oct 30 19:35:37.225492 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:37.225465    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd\": container with ID starting with 847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd not found: ID does not exist" containerID="847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd"
Oct 30 19:35:37.225560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.225506    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd"} err="failed to get container status \"847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd\": rpc error: code = NotFound desc = could not find container \"847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd\": container with ID starting with 847f3b37360a7710f86e5ada6d606aa02d09568b7ee07cfbf6b22d3e882394dd not found: ID does not exist"
Oct 30 19:35:37.241967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.241933    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"]
Oct 30 19:35:37.275121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.257117    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69f3a9-765dc684fd-8rp54"]
Oct 30 19:35:37.514382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:37.514352    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"]
Oct 30 19:35:37.514599 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:35:37.514578    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaff4e94e_4e6a_463e_84a4_60ae69538dd9.slice/crio-0fd82f5dc0480dbf995df9dba95800608cceba89a056621801a312d85a405122 WatchSource:0}: Error finding container 0fd82f5dc0480dbf995df9dba95800608cceba89a056621801a312d85a405122: Status 404 returned error can't find the container with id 0fd82f5dc0480dbf995df9dba95800608cceba89a056621801a312d85a405122
Oct 30 19:35:38.133770 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:38.133731    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc9bd5f-9cac-4cdb-861a-006ec25036fd" path="/var/lib/kubelet/pods/dfc9bd5f-9cac-4cdb-861a-006ec25036fd/volumes"
Oct 30 19:35:38.147945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:38.147911    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" event={"ID":"aff4e94e-4e6a-463e-84a4-60ae69538dd9","Type":"ContainerStarted","Data":"f867b903d8ceaa9d7c461307c0a622a9a30559420ea4998333e2c59a4b489234"}
Oct 30 19:35:38.148135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:38.147957    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" event={"ID":"aff4e94e-4e6a-463e-84a4-60ae69538dd9","Type":"ContainerStarted","Data":"0fd82f5dc0480dbf995df9dba95800608cceba89a056621801a312d85a405122"}
Oct 30 19:35:38.194339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:38.194275    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podStartSLOduration=2.19425525 podStartE2EDuration="2.19425525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:38.191478996 +0000 UTC m=+1720.188394407" watchObservedRunningTime="2026-10-30 19:35:38.19425525 +0000 UTC m=+1720.191170636"
Oct 30 19:35:39.072230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:39.072149    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:43.194944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:43.194849    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"]
Oct 30 19:35:43.195760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:43.195723    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy" containerID="cri-o://d6f91ee3d8a5dfbba58c0fb8deceec31a440fe108848aae3e0e5281478772753" gracePeriod=30
Oct 30 19:35:43.643528 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:43.643483    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:35:44.413665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.413628    6777 patch_prober.go:29] interesting pod/e2e-inf-b-c86696-openshift-default-859496475c-srmqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:35:44.414214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.413708    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:35:44.465719 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.465683    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"]
Oct 30 19:35:44.467165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.467136    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy"
Oct 30 19:35:44.467165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.467164    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61c55b9-72e4-47a6-b8df-1f8ac5ecf799" containerName="istio-proxy"
Oct 30 19:35:44.493247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.493204    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"]
Oct 30 19:35:44.493436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.493332    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.521414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.521379    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8e6762-openshift-default-dockercfg-ns2mw\""
Oct 30 19:35:44.524484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.524454    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.524594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.524490    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.524638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.524603    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.524687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.524663    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.524828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.524796    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.524951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.524932    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8m6\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-kube-api-access-wv8m6\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.525088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.525071    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.525142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.525130    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.525308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.525286    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.626713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.626674    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.626759    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.626819    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.626899    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8m6\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-kube-api-access-wv8m6\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.626996    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.627082    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.627256    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.627290    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.627320    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.627359    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.627378    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.627610    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.627657    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.627750    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.629219    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.629471    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.716032    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8m6\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-kube-api-access-wv8m6\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.727838    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:44.887418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:44.831545    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:45.442384 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:35:45.442350    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8d5446_5b96_41f1_815e_88c0d09b24a0.slice/crio-f87db58e4f032fcc85d7b396ee8732689d7e06d5012dd7ea85797a35b36b0ff4 WatchSource:0}: Error finding container f87db58e4f032fcc85d7b396ee8732689d7e06d5012dd7ea85797a35b36b0ff4: Status 404 returned error can't find the container with id f87db58e4f032fcc85d7b396ee8732689d7e06d5012dd7ea85797a35b36b0ff4
Oct 30 19:35:45.446014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:45.444809    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"]
Oct 30 19:35:45.447504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:45.446376    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:35:45.447504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:45.446448    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:35:45.447504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:45.446481    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:35:45.894503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:45.894465    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn" podUID="2176303e-2380-47ad-81c5-afba1009961d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:46.218492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.218388    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8" podUID="a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:46.219053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.219022    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8" podUID="eaf2532e-7980-44d9-b14e-650e57d53c1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:46.303891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.303843    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" event={"ID":"ba8d5446-5b96-41f1-815e-88c0d09b24a0","Type":"ContainerStarted","Data":"0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5"}
Oct 30 19:35:46.304154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.303923    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" event={"ID":"ba8d5446-5b96-41f1-815e-88c0d09b24a0","Type":"ContainerStarted","Data":"f87db58e4f032fcc85d7b396ee8732689d7e06d5012dd7ea85797a35b36b0ff4"}
Oct 30 19:35:46.305620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.305596    6777 generic.go:356] "Generic (PLEG): container finished" podID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerID="c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b" exitCode=0
Oct 30 19:35:46.305728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.305642    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" event={"ID":"08eb5014-7d21-4071-a25e-67d977c69bd0","Type":"ContainerDied","Data":"c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b"}
Oct 30 19:35:46.306051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.306036    6777 scope.go:122] "RemoveContainer" containerID="c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b"
Oct 30 19:35:46.307852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.307822    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:35:46.307945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.307891    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:35:46.307945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.307924    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:35:46.460510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.460449    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podStartSLOduration=2.46042983 podStartE2EDuration="2.46042983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:46.365127542 +0000 UTC m=+1728.362042928" watchObservedRunningTime="2026-10-30 19:35:46.46042983 +0000 UTC m=+1728.457345212"
Oct 30 19:35:46.832156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.832121    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:46.840802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:46.840765    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:47.081130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.072126    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:47.278139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.278015    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:47.342775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.342736    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" event={"ID":"08eb5014-7d21-4071-a25e-67d977c69bd0","Type":"ContainerStarted","Data":"4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14"}
Oct 30 19:35:47.348373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.344846    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:47.360857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.360769    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:35:47.407052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.406781    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:35:47.520313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.519619    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"]
Oct 30 19:35:47.527461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.527414    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69f3a9-cqrlv"]
Oct 30 19:35:47.925124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.925091    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:35:47.926394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.926364    6777 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:35:47.926525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:47.926423    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 19:35:48.120558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.120511    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8316b0da-7ad2-4fe5-8cef-b47a2289f3e4" path="/var/lib/kubelet/pods/8316b0da-7ad2-4fe5-8cef-b47a2289f3e4/volumes"
Oct 30 19:35:48.382886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.382826    6777 generic.go:356] "Generic (PLEG): container finished" podID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerID="d6f91ee3d8a5dfbba58c0fb8deceec31a440fe108848aae3e0e5281478772753" exitCode=0
Oct 30 19:35:48.383085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.382891    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" event={"ID":"9dcc116f-353d-4ba8-836b-ca158e68bac4","Type":"ContainerDied","Data":"d6f91ee3d8a5dfbba58c0fb8deceec31a440fe108848aae3e0e5281478772753"}
Oct 30 19:35:48.383085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.382950    6777 scope.go:122] "RemoveContainer" containerID="71ef753f4bae2c2cc33b7646002cf92a65e1fc90d3666a0c8a34fc20876231dd"
Oct 30 19:35:48.542746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.542714    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:35:48.691109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.690233    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-envoy\") pod \"9dcc116f-353d-4ba8-836b-ca158e68bac4\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") "
Oct 30 19:35:48.691109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.690329    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-token\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-token\") pod \"9dcc116f-353d-4ba8-836b-ca158e68bac4\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") "
Oct 30 19:35:48.691109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.690460    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-credential-socket\") pod \"9dcc116f-353d-4ba8-836b-ca158e68bac4\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") "
Oct 30 19:35:48.691109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.690555    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-data\") pod \"9dcc116f-353d-4ba8-836b-ca158e68bac4\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") "
Oct 30 19:35:48.691109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.690610    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-certs\") pod \"9dcc116f-353d-4ba8-836b-ca158e68bac4\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") "
Oct 30 19:35:48.691109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.690699    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dcc116f-353d-4ba8-836b-ca158e68bac4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcc116f-353d-4ba8-836b-ca158e68bac4-istiod-ca-cert\") pod \"9dcc116f-353d-4ba8-836b-ca158e68bac4\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") "
Oct 30 19:35:48.691109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.690724    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-socket\") pod \"9dcc116f-353d-4ba8-836b-ca158e68bac4\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") "
Oct 30 19:35:48.691109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.690785    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-kube-api-access-rt7jm\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-kube-api-access-rt7jm\") pod \"9dcc116f-353d-4ba8-836b-ca158e68bac4\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") "
Oct 30 19:35:48.691109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.690819    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-podinfo\") pod \"9dcc116f-353d-4ba8-836b-ca158e68bac4\" (UID: \"9dcc116f-353d-4ba8-836b-ca158e68bac4\") "
Oct 30 19:35:48.693392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.693354    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-data" pod "9dcc116f-353d-4ba8-836b-ca158e68bac4" (UID: "9dcc116f-353d-4ba8-836b-ca158e68bac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:48.693610 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.693582    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-credential-socket" pod "9dcc116f-353d-4ba8-836b-ca158e68bac4" (UID: "9dcc116f-353d-4ba8-836b-ca158e68bac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:48.693916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.693892    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dcc116f-353d-4ba8-836b-ca158e68bac4-istiod-ca-cert" pod "9dcc116f-353d-4ba8-836b-ca158e68bac4" (UID: "9dcc116f-353d-4ba8-836b-ca158e68bac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:48.694141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.694101    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-certs" pod "9dcc116f-353d-4ba8-836b-ca158e68bac4" (UID: "9dcc116f-353d-4ba8-836b-ca158e68bac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:48.694499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.694479    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-socket" pod "9dcc116f-353d-4ba8-836b-ca158e68bac4" (UID: "9dcc116f-353d-4ba8-836b-ca158e68bac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:48.706632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.706593    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-envoy" pod "9dcc116f-353d-4ba8-836b-ca158e68bac4" (UID: "9dcc116f-353d-4ba8-836b-ca158e68bac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:48.706793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.706752    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-podinfo" pod "9dcc116f-353d-4ba8-836b-ca158e68bac4" (UID: "9dcc116f-353d-4ba8-836b-ca158e68bac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:48.709099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.709020    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-token" pod "9dcc116f-353d-4ba8-836b-ca158e68bac4" (UID: "9dcc116f-353d-4ba8-836b-ca158e68bac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:48.715860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.715813    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-kube-api-access-rt7jm" pod "9dcc116f-353d-4ba8-836b-ca158e68bac4" (UID: "9dcc116f-353d-4ba8-836b-ca158e68bac4"). InnerVolumeSpecName "kube-api-access-rt7jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:48.793545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.793508    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:48.793545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.793541    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:48.793545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.793550    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:48.793824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.793559    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcc116f-353d-4ba8-836b-ca158e68bac4-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:48.793824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.793568    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:48.793824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.793581    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt7jm\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-kube-api-access-rt7jm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:48.793824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.793597    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:48.793824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.793608    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:48.793824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.793620    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dcc116f-353d-4ba8-836b-ca158e68bac4-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:48.925357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.925312    6777 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:35:48.926489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:48.925388    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 19:35:49.408728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.408128    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"
Oct 30 19:35:49.410538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.410419    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr" event={"ID":"9dcc116f-353d-4ba8-836b-ca158e68bac4","Type":"ContainerDied","Data":"32254d5eab7b2597aeb3f399988910cb3c57b207c40b3a6f248ea45bd060870c"}
Oct 30 19:35:49.410538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.410495    6777 scope.go:122] "RemoveContainer" containerID="d6f91ee3d8a5dfbba58c0fb8deceec31a440fe108848aae3e0e5281478772753"
Oct 30 19:35:49.475102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.473866    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"]
Oct 30 19:35:49.510083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.504491    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c86696-openshift-default-859496475c-srmqr"]
Oct 30 19:35:49.925236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.925193    6777 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:35:49.925786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.925254    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 19:35:49.992265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.992228    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"]
Oct 30 19:35:49.993785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.993755    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy"
Oct 30 19:35:49.993785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.993784    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy"
Oct 30 19:35:49.994124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.994107    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy"
Oct 30 19:35:49.994294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:49.994276    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy"
Oct 30 19:35:50.004320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.003590    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"]
Oct 30 19:35:50.004320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.003746    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.013557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.013386    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-fd7c77-openshift-default-dockercfg-9t69p\""
Oct 30 19:35:50.202145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.175520    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.202145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.175636    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.202145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.175701    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.202145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.175797    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.202145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.175964    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.202145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.176317    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.202145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.176389    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.202145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.176445    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.202145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.176480    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhw4v\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-kube-api-access-rhw4v\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.254573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.254535    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" path="/var/lib/kubelet/pods/9dcc116f-353d-4ba8-836b-ca158e68bac4/volumes"
Oct 30 19:35:50.279450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.279404    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.290084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.280794    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.291612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.291580    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.291901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.291881    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.292207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.292189    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.292545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.292525    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.292688 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.292673    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.292812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.292800    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.292926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.292909    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhw4v\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-kube-api-access-rhw4v\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.305086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.294326    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.305345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.303757    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.305443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.303992    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.305889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.305864    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.306396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.306372    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.310093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.309853    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.322081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.312607    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.369082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.366775    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhw4v\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-kube-api-access-rhw4v\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.394079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.391192    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.623457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.623416    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:50.947149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.944126    6777 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:35:50.947149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:50.944204    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 19:35:51.177073 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.177028    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:35:51.277690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.277662    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"]
Oct 30 19:35:51.278356 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:35:51.278326    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a01e0ca_b6ee_44c6_be57_f94a25d39f21.slice/crio-25e62c6f99ff2372a2447cdd852da385ee9b67830835dbf417585982979e861e WatchSource:0}: Error finding container 25e62c6f99ff2372a2447cdd852da385ee9b67830835dbf417585982979e861e: Status 404 returned error can't find the container with id 25e62c6f99ff2372a2447cdd852da385ee9b67830835dbf417585982979e861e
Oct 30 19:35:51.281406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.281371    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:35:51.281519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.281455    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:35:51.281519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.281507    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:35:51.474084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.450235    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" event={"ID":"1a01e0ca-b6ee-44c6-be57-f94a25d39f21","Type":"ContainerStarted","Data":"25e62c6f99ff2372a2447cdd852da385ee9b67830835dbf417585982979e861e"}
Oct 30 19:35:51.484655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.484576    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podStartSLOduration=2.484551636 podStartE2EDuration="2.484551636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:51.475954764 +0000 UTC m=+1733.472870151" watchObservedRunningTime="2026-10-30 19:35:51.484551636 +0000 UTC m=+1733.481467022"
Oct 30 19:35:51.623874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.623841    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:51.926195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.926160    6777 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:35:51.926390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:51.926217    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 19:35:52.246114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.246079    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:35:52.333148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.333119    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-plugins-config-volume\") pod \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\" (UID: \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\") "
Oct 30 19:35:52.333327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.333250    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-kube-api-access-66nbj\" (UniqueName: \"kubernetes.io/projected/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-kube-api-access-66nbj\") pod \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\" (UID: \"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0\") "
Oct 30 19:35:52.333583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.333542    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-plugins-config-volume" pod "a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" (UID: "a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:52.334155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.334132    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:52.335635 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.335612    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-kube-api-access-66nbj" pod "a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" (UID: "a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0"). InnerVolumeSpecName "kube-api-access-66nbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:52.336619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.336597    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:35:52.434877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.434776    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf2532e-7980-44d9-b14e-650e57d53c1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2532e-7980-44d9-b14e-650e57d53c1c-plugins-config-volume\") pod \"eaf2532e-7980-44d9-b14e-650e57d53c1c\" (UID: \"eaf2532e-7980-44d9-b14e-650e57d53c1c\") "
Oct 30 19:35:52.435122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.434902    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf2532e-7980-44d9-b14e-650e57d53c1c-kube-api-access-4cfzp\" (UniqueName: \"kubernetes.io/projected/eaf2532e-7980-44d9-b14e-650e57d53c1c-kube-api-access-4cfzp\") pod \"eaf2532e-7980-44d9-b14e-650e57d53c1c\" (UID: \"eaf2532e-7980-44d9-b14e-650e57d53c1c\") "
Oct 30 19:35:52.435307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.435278    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf2532e-7980-44d9-b14e-650e57d53c1c-plugins-config-volume" pod "eaf2532e-7980-44d9-b14e-650e57d53c1c" (UID: "eaf2532e-7980-44d9-b14e-650e57d53c1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:52.435619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.435598    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66nbj\" (UniqueName: \"kubernetes.io/projected/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0-kube-api-access-66nbj\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:52.435684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.435625    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2532e-7980-44d9-b14e-650e57d53c1c-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:52.437105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.437053    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf2532e-7980-44d9-b14e-650e57d53c1c-kube-api-access-4cfzp" pod "eaf2532e-7980-44d9-b14e-650e57d53c1c" (UID: "eaf2532e-7980-44d9-b14e-650e57d53c1c"). InnerVolumeSpecName "kube-api-access-4cfzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:52.484107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.484036    6777 generic.go:356] "Generic (PLEG): container finished" podID="eaf2532e-7980-44d9-b14e-650e57d53c1c" containerID="b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052" exitCode=0
Oct 30 19:35:52.484107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.484102    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"
Oct 30 19:35:52.484340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.484104    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8" event={"ID":"eaf2532e-7980-44d9-b14e-650e57d53c1c","Type":"ContainerDied","Data":"b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052"}
Oct 30 19:35:52.484340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.484226    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8" event={"ID":"eaf2532e-7980-44d9-b14e-650e57d53c1c","Type":"ContainerDied","Data":"acc7586121de3bad6dcb8ec91b7fbb1488d6d03299d716c1b96c1a93fa9221b7"}
Oct 30 19:35:52.484340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.484255    6777 scope.go:122] "RemoveContainer" containerID="b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052"
Oct 30 19:35:52.485911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.485888    6777 generic.go:356] "Generic (PLEG): container finished" podID="a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" containerID="cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954" exitCode=0
Oct 30 19:35:52.486033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.485948    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"
Oct 30 19:35:52.486033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.485950    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8" event={"ID":"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0","Type":"ContainerDied","Data":"cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954"}
Oct 30 19:35:52.486226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.486114    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8" event={"ID":"a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0","Type":"ContainerDied","Data":"e13d08caf64283b00e8d62bba5b58a5ec55813a2884e24315a563ede6f503798"}
Oct 30 19:35:52.488299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.488273    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" event={"ID":"1a01e0ca-b6ee-44c6-be57-f94a25d39f21","Type":"ContainerStarted","Data":"b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b"}
Oct 30 19:35:52.499406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.498873    6777 scope.go:122] "RemoveContainer" containerID="b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052"
Oct 30 19:35:52.499480 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:52.499433    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052\": container with ID starting with b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052 not found: ID does not exist" containerID="b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052"
Oct 30 19:35:52.499480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.499465    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052"} err="failed to get container status \"b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052\": rpc error: code = NotFound desc = could not find container \"b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052\": container with ID starting with b5c0e014efb1702cd642eae98a7692a05b5246ba49d0e568efbc8fa67b90a052 not found: ID does not exist"
Oct 30 19:35:52.499554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.499482    6777 scope.go:122] "RemoveContainer" containerID="cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954"
Oct 30 19:35:52.511946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.511920    6777 scope.go:122] "RemoveContainer" containerID="cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954"
Oct 30 19:35:52.512335 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:35:52.512304    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954\": container with ID starting with cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954 not found: ID does not exist" containerID="cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954"
Oct 30 19:35:52.512417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.512345    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954"} err="failed to get container status \"cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954\": rpc error: code = NotFound desc = could not find container \"cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954\": container with ID starting with cdd6991552ebbf34e51708f331edf99c50015c177f2a049bbbe3e4874c147954 not found: ID does not exist"
Oct 30 19:35:52.517941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.517911    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"]
Oct 30 19:35:52.523695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.523667    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c86696-88fc5bdbb-d8nq8"]
Oct 30 19:35:52.536453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.536415    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"]
Oct 30 19:35:52.537140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.537114    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cfzp\" (UniqueName: \"kubernetes.io/projected/eaf2532e-7980-44d9-b14e-650e57d53c1c-kube-api-access-4cfzp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:52.543029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.543001    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c86696-67f7dffff9-nm7h8"]
Oct 30 19:35:52.632966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.632920    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:52.925223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.925178    6777 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:35:52.925419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:52.925232    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 19:35:53.503876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:53.503695    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:53.504711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:53.504685    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:35:53.929592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:53.929560    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:35:53.930912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:53.930890    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:35:54.118647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:54.118610    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" path="/var/lib/kubelet/pods/a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0/volumes"
Oct 30 19:35:54.119830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:54.119805    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf2532e-7980-44d9-b14e-650e57d53c1c" path="/var/lib/kubelet/pods/eaf2532e-7980-44d9-b14e-650e57d53c1c/volumes"
Oct 30 19:35:55.453033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.452995    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"]
Oct 30 19:35:55.454114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.454084    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy"
Oct 30 19:35:55.454114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.454113    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcc116f-353d-4ba8-836b-ca158e68bac4" containerName="istio-proxy"
Oct 30 19:35:55.454298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.454130    6777 cpu_manager.go:414] "containerMap: removing container" podUID="eaf2532e-7980-44d9-b14e-650e57d53c1c" containerName="payload-pre-processing"
Oct 30 19:35:55.454298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.454138    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf2532e-7980-44d9-b14e-650e57d53c1c" containerName="payload-pre-processing"
Oct 30 19:35:55.454298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.454157    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" containerName="payload-processing"
Oct 30 19:35:55.454298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.454165    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" containerName="payload-processing"
Oct 30 19:35:55.454504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.454378    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e1fddb-9a89-4d9d-a730-7d063e6fa0d0" containerName="payload-processing"
Oct 30 19:35:55.454504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.454411    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf2532e-7980-44d9-b14e-650e57d53c1c" containerName="payload-pre-processing"
Oct 30 19:35:55.460517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.460489    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:35:55.465879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.465851    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"]
Oct 30 19:35:55.476465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.476432    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-fd7c77\""
Oct 30 19:35:55.555355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.555317    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7drw\" (UniqueName: \"kubernetes.io/projected/ba70864a-66bd-44b6-9923-0ac9f00260fd-kube-api-access-k7drw\") pod \"payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn\" (UID: \"ba70864a-66bd-44b6-9923-0ac9f00260fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:35:55.555540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.555527    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba70864a-66bd-44b6-9923-0ac9f00260fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn\" (UID: \"ba70864a-66bd-44b6-9923-0ac9f00260fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:35:55.672128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.671612    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7drw\" (UniqueName: \"kubernetes.io/projected/ba70864a-66bd-44b6-9923-0ac9f00260fd-kube-api-access-k7drw\") pod \"payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn\" (UID: \"ba70864a-66bd-44b6-9923-0ac9f00260fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:35:55.672128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.671821    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba70864a-66bd-44b6-9923-0ac9f00260fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn\" (UID: \"ba70864a-66bd-44b6-9923-0ac9f00260fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:35:55.672919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.672840    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba70864a-66bd-44b6-9923-0ac9f00260fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn\" (UID: \"ba70864a-66bd-44b6-9923-0ac9f00260fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:35:55.837129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.836196    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7drw\" (UniqueName: \"kubernetes.io/projected/ba70864a-66bd-44b6-9923-0ac9f00260fd-kube-api-access-k7drw\") pod \"payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn\" (UID: \"ba70864a-66bd-44b6-9923-0ac9f00260fd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:35:55.901363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:55.899787    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn" podUID="2176303e-2380-47ad-81c5-afba1009961d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:56.079546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.079501    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:35:56.298474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.298442    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"]
Oct 30 19:35:56.298878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.298853    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy" containerID="cri-o://c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69" gracePeriod=30
Oct 30 19:35:56.310642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.310612    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"]
Oct 30 19:35:56.310876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.310846    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy" containerID="cri-o://68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172" gracePeriod=30
Oct 30 19:35:56.391431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.391399    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"]
Oct 30 19:35:56.614438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.614342    6777 generic.go:356] "Generic (PLEG): container finished" podID="2176303e-2380-47ad-81c5-afba1009961d" containerID="2c4be0ea519487ec959908a4e0f3259c59525ca4dae260e248b53e01de0f5217" exitCode=0
Oct 30 19:35:56.615917 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.615884    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn" event={"ID":"2176303e-2380-47ad-81c5-afba1009961d","Type":"ContainerDied","Data":"2c4be0ea519487ec959908a4e0f3259c59525ca4dae260e248b53e01de0f5217"}
Oct 30 19:35:56.627344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.627258    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn" event={"ID":"ba70864a-66bd-44b6-9923-0ac9f00260fd","Type":"ContainerStarted","Data":"4dc8be2fe868d5df2bcca81e64d593a102ffba9ea8a4447a96c5f371ec0648a3"}
Oct 30 19:35:56.824143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.818828    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:35:56.966732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.966637    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2176303e-2380-47ad-81c5-afba1009961d-kube-api-access-p2c8f\" (UniqueName: \"kubernetes.io/projected/2176303e-2380-47ad-81c5-afba1009961d-kube-api-access-p2c8f\") pod \"2176303e-2380-47ad-81c5-afba1009961d\" (UID: \"2176303e-2380-47ad-81c5-afba1009961d\") "
Oct 30 19:35:56.966928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.966771    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2176303e-2380-47ad-81c5-afba1009961d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2176303e-2380-47ad-81c5-afba1009961d-plugins-config-volume\") pod \"2176303e-2380-47ad-81c5-afba1009961d\" (UID: \"2176303e-2380-47ad-81c5-afba1009961d\") "
Oct 30 19:35:56.967248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.967219    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2176303e-2380-47ad-81c5-afba1009961d-plugins-config-volume" pod "2176303e-2380-47ad-81c5-afba1009961d" (UID: "2176303e-2380-47ad-81c5-afba1009961d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:56.967583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.967565    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2176303e-2380-47ad-81c5-afba1009961d-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:56.969175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:56.969146    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2176303e-2380-47ad-81c5-afba1009961d-kube-api-access-p2c8f" pod "2176303e-2380-47ad-81c5-afba1009961d" (UID: "2176303e-2380-47ad-81c5-afba1009961d"). InnerVolumeSpecName "kube-api-access-p2c8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:57.069084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:57.069040    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2c8f\" (UniqueName: \"kubernetes.io/projected/2176303e-2380-47ad-81c5-afba1009961d-kube-api-access-p2c8f\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.637974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:57.637934    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn" event={"ID":"ba70864a-66bd-44b6-9923-0ac9f00260fd","Type":"ContainerStarted","Data":"7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f"}
Oct 30 19:35:57.638489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:57.638010    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:35:57.639566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:57.639537    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"
Oct 30 19:35:57.639708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:57.639581    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn" event={"ID":"2176303e-2380-47ad-81c5-afba1009961d","Type":"ContainerDied","Data":"d2dd0cc49fbedb115514889ee1152598ba945e1715916e9621d08f84d8998572"}
Oct 30 19:35:57.639708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:57.639644    6777 scope.go:122] "RemoveContainer" containerID="2c4be0ea519487ec959908a4e0f3259c59525ca4dae260e248b53e01de0f5217"
Oct 30 19:35:57.678604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:57.678402    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn" podStartSLOduration=2.678380835 podStartE2EDuration="2.678380835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:57.669518742 +0000 UTC m=+1739.666434129" watchObservedRunningTime="2026-10-30 19:35:57.678380835 +0000 UTC m=+1739.675296222"
Oct 30 19:35:57.697413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:57.695803    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"]
Oct 30 19:35:57.701512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:57.701476    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69f3a9-5744d86bc-ljlpn"]
Oct 30 19:35:58.114366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:58.114333    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2176303e-2380-47ad-81c5-afba1009961d" path="/var/lib/kubelet/pods/2176303e-2380-47ad-81c5-afba1009961d/volumes"
Oct 30 19:35:59.825053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:59.825016    6777 scope.go:122] "RemoveContainer" containerID="38b2fe974e2c3412c11259640d0382026cbc1e7bd665d0cddc777e2d7d38c370"
Oct 30 19:35:59.838126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:35:59.838101    6777 scope.go:122] "RemoveContainer" containerID="24ab41d151ffbd81f1b2ed5d1d72778b115381994047ee64be10c82f79e9116b"
Oct 30 19:36:01.646409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.646376    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:36:01.666816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.666791    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:36:01.691822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.691788    6777 generic.go:356] "Generic (PLEG): container finished" podID="11d65393-28cb-433f-9157-0ad2f5e90083" containerID="c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69" exitCode=0
Oct 30 19:36:01.692022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.691847    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"
Oct 30 19:36:01.692022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.691872    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" event={"ID":"11d65393-28cb-433f-9157-0ad2f5e90083","Type":"ContainerDied","Data":"c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69"}
Oct 30 19:36:01.692022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.691936    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln" event={"ID":"11d65393-28cb-433f-9157-0ad2f5e90083","Type":"ContainerDied","Data":"ed3c4d03f47dce9db1605a3c47d07fd016f2ae05819d698023c261c7602dbab3"}
Oct 30 19:36:01.692022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.691971    6777 scope.go:122] "RemoveContainer" containerID="c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69"
Oct 30 19:36:01.694230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.694202    6777 generic.go:356] "Generic (PLEG): container finished" podID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerID="68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172" exitCode=0
Oct 30 19:36:01.694356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.694254    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"
Oct 30 19:36:01.694437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.694401    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" event={"ID":"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24","Type":"ContainerDied","Data":"68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172"}
Oct 30 19:36:01.694498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.694450    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk" event={"ID":"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24","Type":"ContainerDied","Data":"f3896c4814216d6fc4ac7f5326d2ad784b5fe4f6d5feb5b41273e3247d45b916"}
Oct 30 19:36:01.705352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.705332    6777 scope.go:122] "RemoveContainer" containerID="8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496"
Oct 30 19:36:01.722185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.722160    6777 scope.go:122] "RemoveContainer" containerID="c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69"
Oct 30 19:36:01.722593 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:36:01.722565    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69\": container with ID starting with c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69 not found: ID does not exist" containerID="c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69"
Oct 30 19:36:01.722701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.722602    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69"} err="failed to get container status \"c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69\": rpc error: code = NotFound desc = could not find container \"c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69\": container with ID starting with c0d84002c5d744fe7301c052672cded8589c9cb0af1acf478bdb5c3874080c69 not found: ID does not exist"
Oct 30 19:36:01.722701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.722621    6777 scope.go:122] "RemoveContainer" containerID="8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496"
Oct 30 19:36:01.722958 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:36:01.722922    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496\": container with ID starting with 8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496 not found: ID does not exist" containerID="8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496"
Oct 30 19:36:01.723170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.722971    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496"} err="failed to get container status \"8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496\": rpc error: code = NotFound desc = could not find container \"8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496\": container with ID starting with 8e3690bfca2a803174847a30bb901e5b1879321bffb58b73eba7b5cc5f972496 not found: ID does not exist"
Oct 30 19:36:01.723170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.723001    6777 scope.go:122] "RemoveContainer" containerID="68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172"
Oct 30 19:36:01.735010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.734960    6777 scope.go:122] "RemoveContainer" containerID="68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172"
Oct 30 19:36:01.735384 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:36:01.735357    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172\": container with ID starting with 68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172 not found: ID does not exist" containerID="68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172"
Oct 30 19:36:01.735491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.735394    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172"} err="failed to get container status \"68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172\": rpc error: code = NotFound desc = could not find container \"68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172\": container with ID starting with 68ab91816d9c7d3e53db2d75fd2be14f51ee9b10367df3e1c02a1a0ec2a7f172 not found: ID does not exist"
Oct 30 19:36:01.770591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.770557    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-certs\") pod \"11d65393-28cb-433f-9157-0ad2f5e90083\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") "
Oct 30 19:36:01.770783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.770629    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-socket\") pod \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") "
Oct 30 19:36:01.770783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.770692    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-kube-api-access-v4wpj\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-kube-api-access-v4wpj\") pod \"11d65393-28cb-433f-9157-0ad2f5e90083\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") "
Oct 30 19:36:01.770783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.770729    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-credential-socket\") pod \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") "
Oct 30 19:36:01.770997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.770799    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-certs\") pod \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") "
Oct 30 19:36:01.770997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.770839    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-certs" pod "11d65393-28cb-433f-9157-0ad2f5e90083" (UID: "11d65393-28cb-433f-9157-0ad2f5e90083"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.770997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.770849    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istiod-ca-cert\") pod \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") "
Oct 30 19:36:01.770997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.770952    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-envoy\") pod \"11d65393-28cb-433f-9157-0ad2f5e90083\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") "
Oct 30 19:36:01.770997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.770963    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-socket" pod "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" (UID: "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.771355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771001    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d65393-28cb-433f-9157-0ad2f5e90083-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d65393-28cb-433f-9157-0ad2f5e90083-istiod-ca-cert\") pod \"11d65393-28cb-433f-9157-0ad2f5e90083\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") "
Oct 30 19:36:01.771355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771045    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-kube-api-access-tqwv8\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-kube-api-access-tqwv8\") pod \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") "
Oct 30 19:36:01.771355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771101    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-socket\") pod \"11d65393-28cb-433f-9157-0ad2f5e90083\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") "
Oct 30 19:36:01.771355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771217    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-istio-token\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-istio-token\") pod \"11d65393-28cb-433f-9157-0ad2f5e90083\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") "
Oct 30 19:36:01.771355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771253    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-data\") pod \"11d65393-28cb-433f-9157-0ad2f5e90083\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") "
Oct 30 19:36:01.771355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771316    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-data\") pod \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") "
Oct 30 19:36:01.771740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771369    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-envoy\") pod \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") "
Oct 30 19:36:01.771740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771399    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d65393-28cb-433f-9157-0ad2f5e90083-istiod-ca-cert" pod "11d65393-28cb-433f-9157-0ad2f5e90083" (UID: "11d65393-28cb-433f-9157-0ad2f5e90083"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:01.771740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771420    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-token\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-token\") pod \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") "
Oct 30 19:36:01.771740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771439    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11d65393-28cb-433f-9157-0ad2f5e90083-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d65393-28cb-433f-9157-0ad2f5e90083-istio-podinfo\") pod \"11d65393-28cb-433f-9157-0ad2f5e90083\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") "
Oct 30 19:36:01.771740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771477    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-credential-socket\") pod \"11d65393-28cb-433f-9157-0ad2f5e90083\" (UID: \"11d65393-28cb-433f-9157-0ad2f5e90083\") "
Oct 30 19:36:01.771740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771493    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-podinfo\") pod \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\" (UID: \"9ec1d99b-4ec2-4e26-8eed-0aae46a24b24\") "
Oct 30 19:36:01.771740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771683    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-data" pod "11d65393-28cb-433f-9157-0ad2f5e90083" (UID: "11d65393-28cb-433f-9157-0ad2f5e90083"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.772234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771798    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-credential-socket" pod "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" (UID: "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.772454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772428    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-data" pod "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" (UID: "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.772820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771858    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-socket" pod "11d65393-28cb-433f-9157-0ad2f5e90083" (UID: "11d65393-28cb-433f-9157-0ad2f5e90083"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.772820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.771869    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-certs" pod "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" (UID: "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.772820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772724    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.772820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772747    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.772820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772762    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.772820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772775    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.772820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772788    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d65393-28cb-433f-9157-0ad2f5e90083-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.772820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772801    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.772820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772815    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.773577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772829    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.773577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.772825    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-credential-socket" pod "11d65393-28cb-433f-9157-0ad2f5e90083" (UID: "11d65393-28cb-433f-9157-0ad2f5e90083"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.773577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.773151    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istiod-ca-cert" pod "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" (UID: "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:01.774286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.774231    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-kube-api-access-v4wpj" pod "11d65393-28cb-433f-9157-0ad2f5e90083" (UID: "11d65393-28cb-433f-9157-0ad2f5e90083"). InnerVolumeSpecName "kube-api-access-v4wpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:01.774431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.774393    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-istio-token" pod "11d65393-28cb-433f-9157-0ad2f5e90083" (UID: "11d65393-28cb-433f-9157-0ad2f5e90083"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:01.775574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.775548    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-envoy" pod "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" (UID: "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.775921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.775900    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-kube-api-access-tqwv8" pod "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" (UID: "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24"). InnerVolumeSpecName "kube-api-access-tqwv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:01.776046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.776023    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-envoy" pod "11d65393-28cb-433f-9157-0ad2f5e90083" (UID: "11d65393-28cb-433f-9157-0ad2f5e90083"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.776192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.776072    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-token" pod "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" (UID: "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:01.777296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.777270    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-podinfo" pod "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" (UID: "9ec1d99b-4ec2-4e26-8eed-0aae46a24b24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:01.779615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.779588    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d65393-28cb-433f-9157-0ad2f5e90083-istio-podinfo" pod "11d65393-28cb-433f-9157-0ad2f5e90083" (UID: "11d65393-28cb-433f-9157-0ad2f5e90083"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:01.874596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874502    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.874596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874535    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.874596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874550    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqwv8\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-kube-api-access-tqwv8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.874908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874623    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.874908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874638    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.874908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874651    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.874908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874663    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d65393-28cb-433f-9157-0ad2f5e90083-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.874908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874676    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d65393-28cb-433f-9157-0ad2f5e90083-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.874908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874688    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.874908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:01.874699    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4wpj\" (UniqueName: \"kubernetes.io/projected/11d65393-28cb-433f-9157-0ad2f5e90083-kube-api-access-v4wpj\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.021301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:02.021249    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"]
Oct 30 19:36:02.028502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:02.028469    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-nqzln"]
Oct 30 19:36:02.042666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:02.042627    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"]
Oct 30 19:36:02.048365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:02.048332    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-pk7nk"]
Oct 30 19:36:02.127018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:02.126914    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" path="/var/lib/kubelet/pods/11d65393-28cb-433f-9157-0ad2f5e90083/volumes"
Oct 30 19:36:02.127661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:02.127644    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" path="/var/lib/kubelet/pods/9ec1d99b-4ec2-4e26-8eed-0aae46a24b24/volumes"
Oct 30 19:36:08.672754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:08.672717    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:36:09.687664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.687626    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"]
Oct 30 19:36:09.688183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.687838    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy" containerID="cri-o://6bc5662f5d2b5e96112073af4136d372f88ecd9fa388f33f5228d8bbc2426589" gracePeriod=30
Oct 30 19:36:09.848915 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.848871    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"]
Oct 30 19:36:09.850128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850105    6777 cpu_manager.go:414] "containerMap: removing container" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy"
Oct 30 19:36:09.850128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850127    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy"
Oct 30 19:36:09.850329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850145    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy"
Oct 30 19:36:09.850329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850155    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy"
Oct 30 19:36:09.850329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850177    6777 cpu_manager.go:414] "containerMap: removing container" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy"
Oct 30 19:36:09.850329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850186    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy"
Oct 30 19:36:09.850329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850249    6777 cpu_manager.go:414] "containerMap: removing container" podUID="2176303e-2380-47ad-81c5-afba1009961d" containerName="payload-pre-processing"
Oct 30 19:36:09.850329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850257    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2176303e-2380-47ad-81c5-afba1009961d" containerName="payload-pre-processing"
Oct 30 19:36:09.850645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850442    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy"
Oct 30 19:36:09.850645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850463    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="2176303e-2380-47ad-81c5-afba1009961d" containerName="payload-pre-processing"
Oct 30 19:36:09.850645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.850482    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec1d99b-4ec2-4e26-8eed-0aae46a24b24" containerName="istio-proxy"
Oct 30 19:36:09.862374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.862341    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:09.867246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.867210    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-2405fb\"/\"openshift-service-ca.crt\""
Oct 30 19:36:09.867246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.867230    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-2405fb\"/\"kube-root-ca.crt\""
Oct 30 19:36:09.867526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.867312    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-2405fb\"/\"default-dockercfg-56nt5\""
Oct 30 19:36:09.867526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.867393    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-2405fb\"/\"sub-test-model-2405fb-kserve-self-signed-certs\""
Oct 30 19:36:09.871568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.871538    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"]
Oct 30 19:36:09.908869 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.908832    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-home\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:09.909078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.909027    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57t5k\" (UniqueName: \"kubernetes.io/projected/c7669ea1-4f99-4316-be8c-54ce10d66e0a-kube-api-access-57t5k\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:09.909139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.909090    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-model-cache\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:09.909197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.909188    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tls-certs\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:09.909251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.909238    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tmp-dir\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:09.909361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:09.909345    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-dshm\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.011350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.011261    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-model-cache\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.011350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.011325    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tls-certs\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.011549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.011377    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tmp-dir\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.011549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.011438    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-dshm\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.011628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.011557    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-home\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.011759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.011737    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57t5k\" (UniqueName: \"kubernetes.io/projected/c7669ea1-4f99-4316-be8c-54ce10d66e0a-kube-api-access-57t5k\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.011865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.011822    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tmp-dir\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.011930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.011740    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-model-cache\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.012020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.011993    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-home\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.013814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.013791    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-dshm\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.014182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.014163    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tls-certs\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.024345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.024317    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57t5k\" (UniqueName: \"kubernetes.io/projected/c7669ea1-4f99-4316-be8c-54ce10d66e0a-kube-api-access-57t5k\") pod \"sub-test-model-2405fb-kserve-684c4577c9-z8q85\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") " pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.183180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.179078    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:10.506201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.506164    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"]
Oct 30 19:36:10.506355 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:36:10.506317    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7669ea1_4f99_4316_be8c_54ce10d66e0a.slice/crio-1f8df2dab9898a7d10ce01a9e14ce24ed7b61d7f92c635d715bac7c45379bc36 WatchSource:0}: Error finding container 1f8df2dab9898a7d10ce01a9e14ce24ed7b61d7f92c635d715bac7c45379bc36: Status 404 returned error can't find the container with id 1f8df2dab9898a7d10ce01a9e14ce24ed7b61d7f92c635d715bac7c45379bc36
Oct 30 19:36:10.803274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.803173    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" event={"ID":"c7669ea1-4f99-4316-be8c-54ce10d66e0a","Type":"ContainerStarted","Data":"f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326"}
Oct 30 19:36:10.803274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.803221    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" event={"ID":"c7669ea1-4f99-4316-be8c-54ce10d66e0a","Type":"ContainerStarted","Data":"1f8df2dab9898a7d10ce01a9e14ce24ed7b61d7f92c635d715bac7c45379bc36"}
Oct 30 19:36:10.826765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:10.826712    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podStartSLOduration=1.826699028 podStartE2EDuration="1.826699028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:10.824563144 +0000 UTC m=+1752.821478529" watchObservedRunningTime="2026-10-30 19:36:10.826699028 +0000 UTC m=+1752.823614413"
Oct 30 19:36:11.179885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:11.179789    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:13.642865 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:36:13.642828    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:36:14.899840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:14.899790    6777 generic.go:356] "Generic (PLEG): container finished" podID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerID="6bc5662f5d2b5e96112073af4136d372f88ecd9fa388f33f5228d8bbc2426589" exitCode=0
Oct 30 19:36:14.900485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:14.900082    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" event={"ID":"0de2b7b1-50d6-400c-973c-f9e5c076042b","Type":"ContainerDied","Data":"6bc5662f5d2b5e96112073af4136d372f88ecd9fa388f33f5228d8bbc2426589"}
Oct 30 19:36:15.045208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.045180    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:36:15.068741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.068704    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-socket\") pod \"0de2b7b1-50d6-400c-973c-f9e5c076042b\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") "
Oct 30 19:36:15.068933 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.068769    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-token\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-token\") pod \"0de2b7b1-50d6-400c-973c-f9e5c076042b\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") "
Oct 30 19:36:15.068933 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.068826    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-data\") pod \"0de2b7b1-50d6-400c-973c-f9e5c076042b\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") "
Oct 30 19:36:15.068933 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.068861    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-podinfo\") pod \"0de2b7b1-50d6-400c-973c-f9e5c076042b\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") "
Oct 30 19:36:15.069163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.068972    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de2b7b1-50d6-400c-973c-f9e5c076042b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de2b7b1-50d6-400c-973c-f9e5c076042b-istiod-ca-cert\") pod \"0de2b7b1-50d6-400c-973c-f9e5c076042b\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") "
Oct 30 19:36:15.069163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.069019    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-socket" pod "0de2b7b1-50d6-400c-973c-f9e5c076042b" (UID: "0de2b7b1-50d6-400c-973c-f9e5c076042b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:15.069163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.069130    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-credential-socket\") pod \"0de2b7b1-50d6-400c-973c-f9e5c076042b\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") "
Oct 30 19:36:15.069322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.069183    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-data" pod "0de2b7b1-50d6-400c-973c-f9e5c076042b" (UID: "0de2b7b1-50d6-400c-973c-f9e5c076042b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:15.069322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.069217    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-certs\") pod \"0de2b7b1-50d6-400c-973c-f9e5c076042b\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") "
Oct 30 19:36:15.069322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.069295    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-envoy\") pod \"0de2b7b1-50d6-400c-973c-f9e5c076042b\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") "
Oct 30 19:36:15.069500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.069369    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-credential-socket" pod "0de2b7b1-50d6-400c-973c-f9e5c076042b" (UID: "0de2b7b1-50d6-400c-973c-f9e5c076042b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:15.069500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.069385    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-kube-api-access-8nqqm\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-kube-api-access-8nqqm\") pod \"0de2b7b1-50d6-400c-973c-f9e5c076042b\" (UID: \"0de2b7b1-50d6-400c-973c-f9e5c076042b\") "
Oct 30 19:36:15.069606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.069509    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-certs" pod "0de2b7b1-50d6-400c-973c-f9e5c076042b" (UID: "0de2b7b1-50d6-400c-973c-f9e5c076042b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:15.069606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.069552    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de2b7b1-50d6-400c-973c-f9e5c076042b-istiod-ca-cert" pod "0de2b7b1-50d6-400c-973c-f9e5c076042b" (UID: "0de2b7b1-50d6-400c-973c-f9e5c076042b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:15.070296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.070271    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.070382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.070302    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.070382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.070318    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de2b7b1-50d6-400c-973c-f9e5c076042b-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.070382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.070333    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.070382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.070347    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.072187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.072142    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-kube-api-access-8nqqm" pod "0de2b7b1-50d6-400c-973c-f9e5c076042b" (UID: "0de2b7b1-50d6-400c-973c-f9e5c076042b"). InnerVolumeSpecName "kube-api-access-8nqqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:15.072417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.072385    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-podinfo" pod "0de2b7b1-50d6-400c-973c-f9e5c076042b" (UID: "0de2b7b1-50d6-400c-973c-f9e5c076042b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:15.072497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.072458    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-envoy" pod "0de2b7b1-50d6-400c-973c-f9e5c076042b" (UID: "0de2b7b1-50d6-400c-973c-f9e5c076042b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:15.072600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.072583    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-token" pod "0de2b7b1-50d6-400c-973c-f9e5c076042b" (UID: "0de2b7b1-50d6-400c-973c-f9e5c076042b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:15.171796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.171762    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.171796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.171794    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nqqm\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-kube-api-access-8nqqm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.171796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.171805    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.172072 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.171814    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de2b7b1-50d6-400c-973c-f9e5c076042b-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.375399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.375357    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:36:15.914170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.913642    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"
Oct 30 19:36:15.914170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.913676    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz" event={"ID":"0de2b7b1-50d6-400c-973c-f9e5c076042b","Type":"ContainerDied","Data":"00bfcc65836af40ce11ed55351631f5ed5a38a0ac70ce153b1582f37e3e76c7f"}
Oct 30 19:36:15.914170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.913740    6777 scope.go:122] "RemoveContainer" containerID="6bc5662f5d2b5e96112073af4136d372f88ecd9fa388f33f5228d8bbc2426589"
Oct 30 19:36:15.947820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.947789    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"]
Oct 30 19:36:15.952811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:15.952772    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fpplz"]
Oct 30 19:36:16.122661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.122619    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" path="/var/lib/kubelet/pods/0de2b7b1-50d6-400c-973c-f9e5c076042b/volumes"
Oct 30 19:36:16.141888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.140557    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"]
Oct 30 19:36:16.141888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.141852    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy"
Oct 30 19:36:16.141888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.141874    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy"
Oct 30 19:36:16.142374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.142022    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de2b7b1-50d6-400c-973c-f9e5c076042b" containerName="istio-proxy"
Oct 30 19:36:16.142374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.142040    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d65393-28cb-433f-9157-0ad2f5e90083" containerName="istio-proxy"
Oct 30 19:36:16.149440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.147672    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.178591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.178516    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"]
Oct 30 19:36:16.181374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.181337    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-69c573\"/\"kube-root-ca.crt\""
Oct 30 19:36:16.181533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.181450    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-69c573\"/\"openshift-service-ca.crt\""
Oct 30 19:36:16.181800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.181781    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-69c573\"/\"default-dockercfg-gzflj\""
Oct 30 19:36:16.182942 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.182916    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-69c573\"/\"auth-test-model-69c573-kserve-self-signed-certs\""
Oct 30 19:36:16.231991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.231957    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-home\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.232193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.232144    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-dshm\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.232244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.232212    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-tmp-dir\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.232301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.232257    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-model-cache\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.232346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.232322    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqv5b\" (UniqueName: \"kubernetes.io/projected/0faac039-491d-4dd9-aac1-5309b17b599e-kube-api-access-mqv5b\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.232392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.232348    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0faac039-491d-4dd9-aac1-5309b17b599e-tls-certs\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.348191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.340898    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-home\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.348191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.341131    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-dshm\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.348191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.341228    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-tmp-dir\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.348191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.341284    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-model-cache\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.348191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.341363    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqv5b\" (UniqueName: \"kubernetes.io/projected/0faac039-491d-4dd9-aac1-5309b17b599e-kube-api-access-mqv5b\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.348191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.341399    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0faac039-491d-4dd9-aac1-5309b17b599e-tls-certs\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.348191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.341413    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-home\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.348191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.342005    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-tmp-dir\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.348191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.342366    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-model-cache\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.396121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.383304    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0faac039-491d-4dd9-aac1-5309b17b599e-tls-certs\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.396121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.383923    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqv5b\" (UniqueName: \"kubernetes.io/projected/0faac039-491d-4dd9-aac1-5309b17b599e-kube-api-access-mqv5b\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.396121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.386459    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-dshm\") pod \"auth-test-model-69c573-kserve-868574f8f7-fl4b9\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") " pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.465043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.464828    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:16.791963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:16.791910    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"]
Oct 30 19:36:17.064966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:17.064930    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" event={"ID":"0faac039-491d-4dd9-aac1-5309b17b599e","Type":"ContainerStarted","Data":"7c2002022a089e603280a7f3673665bf7bca15e194f29dcc2f1af1bba68c7198"}
Oct 30 19:36:18.313097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:18.290941    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" podStartSLOduration=2.290920303 podStartE2EDuration="2.290920303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:18.280042858 +0000 UTC m=+1760.276958244" watchObservedRunningTime="2026-10-30 19:36:18.290920303 +0000 UTC m=+1760.287835686"
Oct 30 19:36:18.313097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:18.311780    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" event={"ID":"0faac039-491d-4dd9-aac1-5309b17b599e","Type":"ContainerStarted","Data":"4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065"}
Oct 30 19:36:18.466440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:18.465874    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:20.180114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:20.180082    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:20.192766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:20.192738    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:21.192604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:21.192576    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:36:22.027451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:22.027413    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"]
Oct 30 19:36:22.027654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:22.027632    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy" containerID="cri-o://7f22391eefe354f45b1f91e52a6e1980fc19e6c0b535c7ca650894621a0ee868" gracePeriod=30
Oct 30 19:36:25.417873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.417832    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"]
Oct 30 19:36:25.423886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.423847    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.436116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.435312    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"]
Oct 30 19:36:25.438423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.438394    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-0a6e66-openshift-default-dockercfg-9mmqk\""
Oct 30 19:36:25.441387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.441364    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"]
Oct 30 19:36:25.446100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.446078    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.478747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.478711    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"]
Oct 30 19:36:25.519380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519334    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.519586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519460    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.519586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519516    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.519586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519546    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.519767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519636    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.519767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519671    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.519767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519724    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqgm\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-kube-api-access-lrqgm\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.519865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519801    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.519865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519832    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.519955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.519931    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.520015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.520003    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.520052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.520027    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.520120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.520054    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.520167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.520147    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.520207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.520187    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.520316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.520301    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.520355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.520330    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrj4\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-kube-api-access-2nrj4\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.520433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.520414    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622096    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622217    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622279    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.622449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622355    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622400    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.622449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622432    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622460    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622543    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqgm\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-kube-api-access-lrqgm\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622580    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622589    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622633    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622669    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.622775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622670    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.622775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622757    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.623123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622815    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.623123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622847    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.623123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622890    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.623123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622922    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.623123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.622965    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.623123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.623090    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.623462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.623132    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrj4\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-kube-api-access-2nrj4\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.623462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.623336    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.623462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.623365    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.623462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.623405    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.623692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.623654    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.623759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.623745    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.623924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.623899    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.624054 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.623931    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.626154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.626130    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.626520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.626497    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.627643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.627619    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.629397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.629371    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.639697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.639479    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.639697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.639652    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrj4\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-kube-api-access-2nrj4\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.640332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.640306    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.640905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.640881    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqgm\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-kube-api-access-lrqgm\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:25.747484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.747381    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:25.782187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:25.782151    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:26.041938 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:36:26.041901    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3dd082_6412_4cdd_93bd_e7da7cb9ba77.slice/crio-4e3cc31dceab1a9108a17ff03922a673a0cc9a1928c4b136d19398cf63faa11b WatchSource:0}: Error finding container 4e3cc31dceab1a9108a17ff03922a673a0cc9a1928c4b136d19398cf63faa11b: Status 404 returned error can't find the container with id 4e3cc31dceab1a9108a17ff03922a673a0cc9a1928c4b136d19398cf63faa11b
Oct 30 19:36:26.042104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.041954    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"]
Oct 30 19:36:26.045566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.045525    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:36:26.045662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.045616    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:36:26.045701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.045671    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:36:26.113192 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:36:26.108916    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f91cea_9443_4aa1_a1ef_9947c4508cc6.slice/crio-ebd40157330895f05120177a85885b9e497286a9fa6b2add157ada84b5541543 WatchSource:0}: Error finding container ebd40157330895f05120177a85885b9e497286a9fa6b2add157ada84b5541543: Status 404 returned error can't find the container with id ebd40157330895f05120177a85885b9e497286a9fa6b2add157ada84b5541543
Oct 30 19:36:26.124031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.123998    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:36:26.124281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.124263    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:36:26.125297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.125271    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:36:26.138997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.135044    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"]
Oct 30 19:36:26.244035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.242997    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" event={"ID":"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77","Type":"ContainerStarted","Data":"5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83"}
Oct 30 19:36:26.244035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.243050    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" event={"ID":"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77","Type":"ContainerStarted","Data":"4e3cc31dceab1a9108a17ff03922a673a0cc9a1928c4b136d19398cf63faa11b"}
Oct 30 19:36:26.247432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.247396    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" event={"ID":"65f91cea-9443-4aa1-a1ef-9947c4508cc6","Type":"ContainerStarted","Data":"ebd40157330895f05120177a85885b9e497286a9fa6b2add157ada84b5541543"}
Oct 30 19:36:26.272126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.268714    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" podStartSLOduration=1.26869673 podStartE2EDuration="1.26869673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:26.266783051 +0000 UTC m=+1768.263698467" watchObservedRunningTime="2026-10-30 19:36:26.26869673 +0000 UTC m=+1768.265612322"
Oct 30 19:36:26.465905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.465825    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:26.478852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.478819    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:26.751170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:26.749308    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:27.312991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.312901    6777 generic.go:356] "Generic (PLEG): container finished" podID="1371c1da-8673-4248-880f-f99c6712a1e3" containerID="7f22391eefe354f45b1f91e52a6e1980fc19e6c0b535c7ca650894621a0ee868" exitCode=0
Oct 30 19:36:27.312991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.312972    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" event={"ID":"1371c1da-8673-4248-880f-f99c6712a1e3","Type":"ContainerDied","Data":"7f22391eefe354f45b1f91e52a6e1980fc19e6c0b535c7ca650894621a0ee868"}
Oct 30 19:36:27.315326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.315300    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" event={"ID":"65f91cea-9443-4aa1-a1ef-9947c4508cc6","Type":"ContainerStarted","Data":"371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04"}
Oct 30 19:36:27.331631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.331602    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:36:27.357085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.348280    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" podStartSLOduration=2.348260943 podStartE2EDuration="2.348260943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:27.344205914 +0000 UTC m=+1769.341121298" watchObservedRunningTime="2026-10-30 19:36:27.348260943 +0000 UTC m=+1769.345176329"
Oct 30 19:36:27.604374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.604290    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:36:27.729739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.729696    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-socket\") pod \"1371c1da-8673-4248-880f-f99c6712a1e3\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") "
Oct 30 19:36:27.730040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.730021    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-envoy\") pod \"1371c1da-8673-4248-880f-f99c6712a1e3\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") "
Oct 30 19:36:27.730229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.730211    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-certs\") pod \"1371c1da-8673-4248-880f-f99c6712a1e3\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") "
Oct 30 19:36:27.730399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.730383    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1371c1da-8673-4248-880f-f99c6712a1e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371c1da-8673-4248-880f-f99c6712a1e3-istio-podinfo\") pod \"1371c1da-8673-4248-880f-f99c6712a1e3\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") "
Oct 30 19:36:27.730573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.730556    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-data\") pod \"1371c1da-8673-4248-880f-f99c6712a1e3\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") "
Oct 30 19:36:27.748927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.730815    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-kube-api-access-7fzpk\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-kube-api-access-7fzpk\") pod \"1371c1da-8673-4248-880f-f99c6712a1e3\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") "
Oct 30 19:36:27.917847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.890377    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-credential-socket\") pod \"1371c1da-8673-4248-880f-f99c6712a1e3\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") "
Oct 30 19:36:27.917847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.890446    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-istio-token\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-istio-token\") pod \"1371c1da-8673-4248-880f-f99c6712a1e3\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") "
Oct 30 19:36:27.917847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.890489    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1371c1da-8673-4248-880f-f99c6712a1e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371c1da-8673-4248-880f-f99c6712a1e3-istiod-ca-cert\") pod \"1371c1da-8673-4248-880f-f99c6712a1e3\" (UID: \"1371c1da-8673-4248-880f-f99c6712a1e3\") "
Oct 30 19:36:27.943082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.748537    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-data" pod "1371c1da-8673-4248-880f-f99c6712a1e3" (UID: "1371c1da-8673-4248-880f-f99c6712a1e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:27.943082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.750422    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:36:27.943082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.923431    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 19:36:27.966545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.961282    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-envoy" pod "1371c1da-8673-4248-880f-f99c6712a1e3" (UID: "1371c1da-8673-4248-880f-f99c6712a1e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:28.001113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.000193    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-credential-socket" pod "1371c1da-8673-4248-880f-f99c6712a1e3" (UID: "1371c1da-8673-4248-880f-f99c6712a1e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.022360    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.022451    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.022467    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.800186    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-certs" pod "1371c1da-8673-4248-880f-f99c6712a1e3" (UID: "1371c1da-8673-4248-880f-f99c6712a1e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.859334    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.022551    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.902291    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-socket" pod "1371c1da-8673-4248-880f-f99c6712a1e3" (UID: "1371c1da-8673-4248-880f-f99c6712a1e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.923316    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1371c1da-8673-4248-880f-f99c6712a1e3-istiod-ca-cert" pod "1371c1da-8673-4248-880f-f99c6712a1e3" (UID: "1371c1da-8673-4248-880f-f99c6712a1e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:27.786610    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:28.049083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.031763    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-istio-token" pod "1371c1da-8673-4248-880f-f99c6712a1e3" (UID: "1371c1da-8673-4248-880f-f99c6712a1e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:28.101140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.087372    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-kube-api-access-7fzpk" pod "1371c1da-8673-4248-880f-f99c6712a1e3" (UID: "1371c1da-8673-4248-880f-f99c6712a1e3"). InnerVolumeSpecName "kube-api-access-7fzpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:28.194947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.145664    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.194947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.145692    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fzpk\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-kube-api-access-7fzpk\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.194947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.145705    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1371c1da-8673-4248-880f-f99c6712a1e3-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.194947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.145715    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371c1da-8673-4248-880f-f99c6712a1e3-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.194947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.145725    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371c1da-8673-4248-880f-f99c6712a1e3-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.194947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.155644    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1371c1da-8673-4248-880f-f99c6712a1e3-istio-podinfo" pod "1371c1da-8673-4248-880f-f99c6712a1e3" (UID: "1371c1da-8673-4248-880f-f99c6712a1e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:28.247614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.247576    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371c1da-8673-4248-880f-f99c6712a1e3-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.327793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.327765    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"
Oct 30 19:36:28.327989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.327797    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq" event={"ID":"1371c1da-8673-4248-880f-f99c6712a1e3","Type":"ContainerDied","Data":"2bac3315d6657fd8aafe43d2aef190461e2d8b60567f00f27cfa4202c8545b12"}
Oct 30 19:36:28.327989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.327853    6777 scope.go:122] "RemoveContainer" containerID="7f22391eefe354f45b1f91e52a6e1980fc19e6c0b535c7ca650894621a0ee868"
Oct 30 19:36:28.426023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.425978    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"]
Oct 30 19:36:28.430441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.430407    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-d2qvq"]
Oct 30 19:36:28.748007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.747966    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:36:28.748557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.748023    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 19:36:28.795567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.795525    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:36:28.795840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:28.795804    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 19:36:29.752527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:29.752488    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:36:29.758186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:29.758139    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 19:36:29.794085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:29.793532    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:36:29.794085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:29.793597    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 19:36:30.117239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.117207    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" path="/var/lib/kubelet/pods/1371c1da-8673-4248-880f-f99c6712a1e3/volumes"
Oct 30 19:36:30.514866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.514773    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"]
Oct 30 19:36:30.515756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.515733    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy"
Oct 30 19:36:30.515756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.515756    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy"
Oct 30 19:36:30.516010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.515993    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1371c1da-8673-4248-880f-f99c6712a1e3" containerName="istio-proxy"
Oct 30 19:36:30.532953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.532922    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"]
Oct 30 19:36:30.533600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.532996    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.545965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.545927    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.546149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.545998    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.546149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.546053    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.546300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.546156    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.546300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.546201    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.546421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.546335    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5g7l\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-kube-api-access-q5g7l\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.546483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.546390    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.546483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.546478    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b316199b-90ea-49c2-92f9-31a270b16d4a-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.546603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.546535    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.648223    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.648311    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.648357    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.648493    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5g7l\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-kube-api-access-q5g7l\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.648553    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.648604    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b316199b-90ea-49c2-92f9-31a270b16d4a-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.648671    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.648843    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.648894    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.649419    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.673084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.649691    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.704469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.699131    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.704469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.700416    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b316199b-90ea-49c2-92f9-31a270b16d4a-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.704730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.704498    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.704730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.704628    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.707444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.707347    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.715593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.715566    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.715835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.715807    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5g7l\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-kube-api-access-q5g7l\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:30.774552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.757380    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:36:30.774552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.757443    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 19:36:30.786683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.784944    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:36:30.786683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.785004    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 19:36:30.919104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:30.917406    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:31.616194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:31.616141    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"]
Oct 30 19:36:31.616890 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:36:31.616863    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb316199b_90ea_49c2_92f9_31a270b16d4a.slice/crio-9f8f2d10972336214ebdcbcb778757807154ab6dd53eacd3cc12d7ae5706a0e4 WatchSource:0}: Error finding container 9f8f2d10972336214ebdcbcb778757807154ab6dd53eacd3cc12d7ae5706a0e4: Status 404 returned error can't find the container with id 9f8f2d10972336214ebdcbcb778757807154ab6dd53eacd3cc12d7ae5706a0e4
Oct 30 19:36:31.619949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:31.619917    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:36:31.620012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:31.619990    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:36:31.620065 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:31.620020    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:36:31.747972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:31.747938    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:36:31.752180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:31.748013    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 19:36:31.792091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:31.784740    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:36:31.792091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:31.784811    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 19:36:32.517587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:32.517540    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" event={"ID":"b316199b-90ea-49c2-92f9-31a270b16d4a","Type":"ContainerStarted","Data":"f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb"}
Oct 30 19:36:32.517801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:32.517657    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" event={"ID":"b316199b-90ea-49c2-92f9-31a270b16d4a","Type":"ContainerStarted","Data":"9f8f2d10972336214ebdcbcb778757807154ab6dd53eacd3cc12d7ae5706a0e4"}
Oct 30 19:36:32.545570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:32.545509    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podStartSLOduration=2.545496682 podStartE2EDuration="2.545496682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:32.543314129 +0000 UTC m=+1774.540229513" watchObservedRunningTime="2026-10-30 19:36:32.545496682 +0000 UTC m=+1774.542412067"
Oct 30 19:36:32.777338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:32.777236    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:36:32.777667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:32.777633    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 19:36:32.789420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:32.789375    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:36:32.789748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:32.789717    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 19:36:32.921866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:32.921830    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:32.928076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:32.928031    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:33.525956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:33.525924    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:33.527136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:33.527110    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:36:33.754235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:33.754203    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:33.754603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:33.754582    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:33.755241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:33.755222    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:36:33.787640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:33.787609    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:33.787912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:33.787894    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:33.788600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:33.788582    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:36:37.091475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.091440    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"]
Oct 30 19:36:37.096804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.096781    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:37.105250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.105219    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"]
Oct 30 19:36:37.105541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.105517    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-1b1531\""
Oct 30 19:36:37.128703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.128669    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"]
Oct 30 19:36:37.133794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.133773    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:37.140907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.140880    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-1b1531-dockercfg-t5xjf\""
Oct 30 19:36:37.142693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.142667    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"]
Oct 30 19:36:37.238090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.232346    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8401cdc-e5c5-4c7a-8f24-078287a136a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5\" (UID: \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:37.238090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.232589    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjsm\" (UniqueName: \"kubernetes.io/projected/63b33f32-ffbe-4bbc-b34e-a439325206ab-kube-api-access-gsjsm\") pod \"payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89\" (UID: \"63b33f32-ffbe-4bbc-b34e-a439325206ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:37.238090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.232840    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdr2\" (UniqueName: \"kubernetes.io/projected/e8401cdc-e5c5-4c7a-8f24-078287a136a7-kube-api-access-4pdr2\") pod \"payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5\" (UID: \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:37.238090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.233135    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b33f32-ffbe-4bbc-b34e-a439325206ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89\" (UID: \"63b33f32-ffbe-4bbc-b34e-a439325206ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:37.341082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.338667    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjsm\" (UniqueName: \"kubernetes.io/projected/63b33f32-ffbe-4bbc-b34e-a439325206ab-kube-api-access-gsjsm\") pod \"payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89\" (UID: \"63b33f32-ffbe-4bbc-b34e-a439325206ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:37.341082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.338880    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdr2\" (UniqueName: \"kubernetes.io/projected/e8401cdc-e5c5-4c7a-8f24-078287a136a7-kube-api-access-4pdr2\") pod \"payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5\" (UID: \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:37.341082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.339076    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b33f32-ffbe-4bbc-b34e-a439325206ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89\" (UID: \"63b33f32-ffbe-4bbc-b34e-a439325206ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:37.341082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.339128    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8401cdc-e5c5-4c7a-8f24-078287a136a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5\" (UID: \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:37.384133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.356357    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b33f32-ffbe-4bbc-b34e-a439325206ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89\" (UID: \"63b33f32-ffbe-4bbc-b34e-a439325206ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:37.384133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.380923    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8401cdc-e5c5-4c7a-8f24-078287a136a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5\" (UID: \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:37.458028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.455555    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjsm\" (UniqueName: \"kubernetes.io/projected/63b33f32-ffbe-4bbc-b34e-a439325206ab-kube-api-access-gsjsm\") pod \"payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89\" (UID: \"63b33f32-ffbe-4bbc-b34e-a439325206ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:37.458028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.456119    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdr2\" (UniqueName: \"kubernetes.io/projected/e8401cdc-e5c5-4c7a-8f24-078287a136a7-kube-api-access-4pdr2\") pod \"payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5\" (UID: \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:37.458028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.456583    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:37.719096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.718960    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:37.819901 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:36:37.819861    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8401cdc_e5c5_4c7a_8f24_078287a136a7.slice/crio-e0833d27ca7c8f20dc936c15e283b89df738b3ef49e605609e9e6356b5efa8ad WatchSource:0}: Error finding container e0833d27ca7c8f20dc936c15e283b89df738b3ef49e605609e9e6356b5efa8ad: Status 404 returned error can't find the container with id e0833d27ca7c8f20dc936c15e283b89df738b3ef49e605609e9e6356b5efa8ad
Oct 30 19:36:37.823252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:37.823210    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"]
Oct 30 19:36:38.028356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:38.028326    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"]
Oct 30 19:36:38.028883 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:36:38.028858    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b33f32_ffbe_4bbc_b34e_a439325206ab.slice/crio-e101cd10030b805c1ad0002c90447ff7005d5d131d9481d103b4faaf8e713cf2 WatchSource:0}: Error finding container e101cd10030b805c1ad0002c90447ff7005d5d131d9481d103b4faaf8e713cf2: Status 404 returned error can't find the container with id e101cd10030b805c1ad0002c90447ff7005d5d131d9481d103b4faaf8e713cf2
Oct 30 19:36:38.582378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:38.582340    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5" event={"ID":"e8401cdc-e5c5-4c7a-8f24-078287a136a7","Type":"ContainerStarted","Data":"460a65234e747931ccc758234e67dfde453038fd2728104d097d8e5bea0ec2e9"}
Oct 30 19:36:38.582846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:38.582400    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5" event={"ID":"e8401cdc-e5c5-4c7a-8f24-078287a136a7","Type":"ContainerStarted","Data":"e0833d27ca7c8f20dc936c15e283b89df738b3ef49e605609e9e6356b5efa8ad"}
Oct 30 19:36:38.582846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:38.582426    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:38.584144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:38.584115    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89" event={"ID":"63b33f32-ffbe-4bbc-b34e-a439325206ab","Type":"ContainerStarted","Data":"5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6"}
Oct 30 19:36:38.584259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:38.584170    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89" event={"ID":"63b33f32-ffbe-4bbc-b34e-a439325206ab","Type":"ContainerStarted","Data":"e101cd10030b805c1ad0002c90447ff7005d5d131d9481d103b4faaf8e713cf2"}
Oct 30 19:36:38.584259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:38.584204    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:38.604863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:38.604811    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5" podStartSLOduration=1.604796398 podStartE2EDuration="1.604796398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:38.602040785 +0000 UTC m=+1780.598956171" watchObservedRunningTime="2026-10-30 19:36:38.604796398 +0000 UTC m=+1780.601711783"
Oct 30 19:36:38.628399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:38.628346    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89" podStartSLOduration=1.62833177 podStartE2EDuration="1.62833177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:38.625217208 +0000 UTC m=+1780.622132595" watchObservedRunningTime="2026-10-30 19:36:38.62833177 +0000 UTC m=+1780.625247154"
Oct 30 19:36:43.642337 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:36:43.642295    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:36:49.604870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:49.604833    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:36:49.605532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:49.604929    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:36:51.634744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.634710    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"]
Oct 30 19:36:51.639639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.639606    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.646299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.646252    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e7123\"/\"openshift-service-ca.crt\""
Oct 30 19:36:51.648099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.648003    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e7123\"/\"kube-root-ca.crt\""
Oct 30 19:36:51.648099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.648033    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e7123\"/\"default-dockercfg-hsdpc\""
Oct 30 19:36:51.648855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.648827    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"]
Oct 30 19:36:51.648969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.648834    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e7123\"/\"sub-test-model-1e7123-kserve-self-signed-certs\""
Oct 30 19:36:51.705031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.704996    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tls-certs\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.705262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.705086    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tmp-dir\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.705262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.705244    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-home\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.705396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.705299    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stk4d\" (UniqueName: \"kubernetes.io/projected/bfae7bd8-8b51-4867-ac02-33beb7c228f9-kube-api-access-stk4d\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.705479 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.705465    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-model-cache\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.705577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.705559    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-dshm\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.806998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.806945    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-dshm\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.807224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.807207    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tls-certs\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.807320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.807297    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tmp-dir\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.807526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.807389    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-home\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.807526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.807434    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stk4d\" (UniqueName: \"kubernetes.io/projected/bfae7bd8-8b51-4867-ac02-33beb7c228f9-kube-api-access-stk4d\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.807655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.807550    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-model-cache\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.807819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.807781    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tmp-dir\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.807988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.807967    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-model-cache\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.809593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.808218    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-home\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.819820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.810243    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tls-certs\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.841914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.841859    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-dshm\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.852459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.852384    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stk4d\" (UniqueName: \"kubernetes.io/projected/bfae7bd8-8b51-4867-ac02-33beb7c228f9-kube-api-access-stk4d\") pod \"sub-test-model-1e7123-kserve-59c74fc4cf-srbks\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") " pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:51.971258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:51.970877    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:36:52.317173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:52.317137    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"]
Oct 30 19:36:52.327268 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:36:52.327228    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfae7bd8_8b51_4867_ac02_33beb7c228f9.slice/crio-42294f7f81d4a0249ca849048e3b8fb667085b1853791b71af012a5a42fda4d8 WatchSource:0}: Error finding container 42294f7f81d4a0249ca849048e3b8fb667085b1853791b71af012a5a42fda4d8: Status 404 returned error can't find the container with id 42294f7f81d4a0249ca849048e3b8fb667085b1853791b71af012a5a42fda4d8
Oct 30 19:36:52.723561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:52.723470    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" event={"ID":"bfae7bd8-8b51-4867-ac02-33beb7c228f9","Type":"ContainerStarted","Data":"f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7"}
Oct 30 19:36:52.723561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:52.723519    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" event={"ID":"bfae7bd8-8b51-4867-ac02-33beb7c228f9","Type":"ContainerStarted","Data":"42294f7f81d4a0249ca849048e3b8fb667085b1853791b71af012a5a42fda4d8"}
Oct 30 19:36:52.744448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:52.744392    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podStartSLOduration=1.7443779209999999 podStartE2EDuration="1.744377921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:52.743804595 +0000 UTC m=+1794.740719979" watchObservedRunningTime="2026-10-30 19:36:52.744377921 +0000 UTC m=+1794.741293304"
Oct 30 19:36:52.974251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:36:52.974152    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:37:00.014843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.014808    6777 scope.go:122] "RemoveContainer" containerID="347fbe097f3fb3a6717a3a117bb98938c8d9696ce03ef638d5c7b316dc017f8b"
Oct 30 19:37:00.032977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.032555    6777 scope.go:122] "RemoveContainer" containerID="abfba7f2b790fae6c488f80c2474ae90b11f27826706213dbebe659e2b66034d"
Oct 30 19:37:00.804622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.804584    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"]
Oct 30 19:37:00.811562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.811536    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:00.821986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.821960    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:00.822292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.821970    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"]
Oct 30 19:37:00.822444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.822043    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:37:00.883637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.883597    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fd39f4-sgch6\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:00.883843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.883655    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bvt\" (UniqueName: \"kubernetes.io/projected/90f30eef-03fc-46a7-957a-c9ea17d1f1af-kube-api-access-82bvt\") pod \"maas-api-revoke-sub-01fd39f4-sgch6\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:00.884015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.883990    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fd39f4-sgch6\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:00.985572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.985536    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fd39f4-sgch6\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:00.985897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.985875    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fd39f4-sgch6\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:00.985979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.985908    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bvt\" (UniqueName: \"kubernetes.io/projected/90f30eef-03fc-46a7-957a-c9ea17d1f1af-kube-api-access-82bvt\") pod \"maas-api-revoke-sub-01fd39f4-sgch6\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:00.986808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.986773    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fd39f4-sgch6\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:00.989005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:00.988980    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fd39f4-sgch6\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:01.021511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:01.016785    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bvt\" (UniqueName: \"kubernetes.io/projected/90f30eef-03fc-46a7-957a-c9ea17d1f1af-kube-api-access-82bvt\") pod \"maas-api-revoke-sub-01fd39f4-sgch6\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:01.150665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:01.150576    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:02.034809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:01.976748    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:37:02.460006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:02.459968    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"]
Oct 30 19:37:02.499198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:02.499147    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:37:02.830458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:02.830429    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:37:02.935528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:02.935411    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:02.937898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:02.936180    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:37:03.082081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.070336    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mlz74" podUID="0c45319f-1d63-435a-85b6-5e49094db5be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.301093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.294283    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:37:03.309102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.308569    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.334160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.333576    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6" event={"ID":"90f30eef-03fc-46a7-957a-c9ea17d1f1af","Type":"ContainerStarted","Data":"dd75e6d9c4248959aa5a01cb74b57449d356307b758e5b409895b79b7673d982"}
Oct 30 19:37:03.334160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.333701    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.350007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.349460    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.555138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.552550    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:37:03.555138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.552994    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:37:03.587789 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.587740    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.914721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.914618    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.947211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.947162    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:37:03.947884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.947846    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:37:03.961446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.961403    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.962028 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.961994    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:37:03.964108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.962812    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:37:03.995684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:03.995637    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.083189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.083144    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.193436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.193327    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.333082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.301396    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.403241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.361679    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.540457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.334585    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.540457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.479875    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.540457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.517754    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded"
Oct 30 19:37:04.596829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.595155    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.704085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.696869    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:37:04.744650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.744574    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:37:04.744914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.744897    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:37:04.745091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.745076    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:37:04.746721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.746694    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:37:04.746839 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.746746    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:37:04.750488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.750459    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6" event={"ID":"90f30eef-03fc-46a7-957a-c9ea17d1f1af","Type":"ContainerStarted","Data":"1c546cea20f54e0aa56a0acd5c9f14fa069d66ef595b9e044091750f36cc7a77"}
Oct 30 19:37:04.858085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.856723    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:37:04.947461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:04.947081    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:37:05.046085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:05.036225    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.268341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:05.036257    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:37:05.310543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:05.310442    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:37:05.330147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:05.288380    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.450083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:05.422961    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.450083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:05.423143    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:37:05.450083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:05.431744    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:37:05.624019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:05.623903    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.732081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:05.731122    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.042252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.042198    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:37:06.138316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.134608    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.161168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.140655    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.212082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.211131    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.331135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.326177    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.401156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.381605    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.441530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.441108    6777 generic.go:356] "Generic (PLEG): container finished" podID="90f30eef-03fc-46a7-957a-c9ea17d1f1af" containerID="1c546cea20f54e0aa56a0acd5c9f14fa069d66ef595b9e044091750f36cc7a77" exitCode=0
Oct 30 19:37:06.445448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.444606    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:37:06.445448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.444715    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6" event={"ID":"90f30eef-03fc-46a7-957a-c9ea17d1f1af","Type":"ContainerDied","Data":"1c546cea20f54e0aa56a0acd5c9f14fa069d66ef595b9e044091750f36cc7a77"}
Oct 30 19:37:06.627639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.625689    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.734388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.734347    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.858922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:06.858392    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.041092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.037639    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:37:07.200083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.157176    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.269244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.250051    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:37:07.269244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.252438    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.332147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.330895    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.332147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.331492    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:37:07.452514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.441154    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.453020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.452993    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:37:07.528082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.522640    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.576084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.575151    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:37:07.647684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.646034    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.647684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.646605    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:37:07.728958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.728916    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.942757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.935769    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.942757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.937007    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:37:07.942757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.937142    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.968791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:07.968171    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:08.181148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.082894    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:37:08.181148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.173345    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:08.325647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.290241    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:37:08.325647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.290303    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:37:08.325647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.291309    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:08.325647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.322203    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:08.469147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.458113    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.546557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.546510    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f30eef-03fc-46a7-957a-c9ea17d1f1af-kube-api-access-82bvt\" (UniqueName: \"kubernetes.io/projected/90f30eef-03fc-46a7-957a-c9ea17d1f1af-kube-api-access-82bvt\") pod \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") "
Oct 30 19:37:08.546978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.546946    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-openshift-ca\") pod \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") "
Oct 30 19:37:08.547498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.547476    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-xks-ca\") pod \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\" (UID: \"90f30eef-03fc-46a7-957a-c9ea17d1f1af\") "
Oct 30 19:37:08.607232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.599533    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-openshift-ca" pod "90f30eef-03fc-46a7-957a-c9ea17d1f1af" (UID: "90f30eef-03fc-46a7-957a-c9ea17d1f1af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:08.646717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.646660    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f30eef-03fc-46a7-957a-c9ea17d1f1af-kube-api-access-82bvt" pod "90f30eef-03fc-46a7-957a-c9ea17d1f1af" (UID: "90f30eef-03fc-46a7-957a-c9ea17d1f1af"). InnerVolumeSpecName "kube-api-access-82bvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:08.705089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.691190    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82bvt\" (UniqueName: \"kubernetes.io/projected/90f30eef-03fc-46a7-957a-c9ea17d1f1af-kube-api-access-82bvt\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:08.706935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.706901    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:08.744525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.744419    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:08.746038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.746011    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:37:08.834186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.757403    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-xks-ca" pod "90f30eef-03fc-46a7-957a-c9ea17d1f1af" (UID: "90f30eef-03fc-46a7-957a-c9ea17d1f1af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:08.847183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.837774    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:37:08.847183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.843315    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f30eef-03fc-46a7-957a-c9ea17d1f1af-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:08.957080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:08.945609    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:37:09.067575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:09.067544    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"
Oct 30 19:37:09.097137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:09.095314    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6" event={"ID":"90f30eef-03fc-46a7-957a-c9ea17d1f1af","Type":"ContainerDied","Data":"dd75e6d9c4248959aa5a01cb74b57449d356307b758e5b409895b79b7673d982"}
Oct 30 19:37:09.097137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:09.095379    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd75e6d9c4248959aa5a01cb74b57449d356307b758e5b409895b79b7673d982"
Oct 30 19:37:13.687730 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:37:13.687682    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:37:16.779962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.779897    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"]
Oct 30 19:37:16.781538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.781511    6777 cpu_manager.go:414] "containerMap: removing container" podUID="90f30eef-03fc-46a7-957a-c9ea17d1f1af" containerName="revoke-keys"
Oct 30 19:37:16.781619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.781541    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f30eef-03fc-46a7-957a-c9ea17d1f1af" containerName="revoke-keys"
Oct 30 19:37:16.781865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.781836    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f30eef-03fc-46a7-957a-c9ea17d1f1af" containerName="revoke-keys"
Oct 30 19:37:16.820838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.820807    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"]
Oct 30 19:37:16.821041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.820932    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:16.826605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.826582    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:37:16.826605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.826596    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:16.841522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.841493    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:16.841631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.841533    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:16.841689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.841668    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzc8h\" (UniqueName: \"kubernetes.io/projected/d03716bc-386f-4728-9b5a-7b28cf5b93d7-kube-api-access-dzc8h\") pod \"maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:16.943441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.943411    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:16.943619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.943450    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:16.943619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.943495    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzc8h\" (UniqueName: \"kubernetes.io/projected/d03716bc-386f-4728-9b5a-7b28cf5b93d7-kube-api-access-dzc8h\") pod \"maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:16.944218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.944177    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:16.946341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.946316    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:16.959243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:16.959219    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzc8h\" (UniqueName: \"kubernetes.io/projected/d03716bc-386f-4728-9b5a-7b28cf5b93d7-kube-api-access-dzc8h\") pod \"maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:17.182500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:17.182429    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:17.599863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:17.599832    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"]
Oct 30 19:37:17.601312 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:37:17.600572    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03716bc_386f_4728_9b5a_7b28cf5b93d7.slice/crio-56b2b438d118a5fac3143e1ff3804510b473458b4f48d1947264019376a8851c WatchSource:0}: Error finding container 56b2b438d118a5fac3143e1ff3804510b473458b4f48d1947264019376a8851c: Status 404 returned error can't find the container with id 56b2b438d118a5fac3143e1ff3804510b473458b4f48d1947264019376a8851c
Oct 30 19:37:18.169442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:18.169404    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2" event={"ID":"d03716bc-386f-4728-9b5a-7b28cf5b93d7","Type":"ContainerStarted","Data":"adfcc77d2983a298328761504c870d3f2749517f11d83aec75301e6a0522051b"}
Oct 30 19:37:18.169840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:18.169457    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2" event={"ID":"d03716bc-386f-4728-9b5a-7b28cf5b93d7","Type":"ContainerStarted","Data":"56b2b438d118a5fac3143e1ff3804510b473458b4f48d1947264019376a8851c"}
Oct 30 19:37:18.224357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:18.224295    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2" podStartSLOduration=2.224276925 podStartE2EDuration="2.224276925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:18.218244036 +0000 UTC m=+1820.215159420" watchObservedRunningTime="2026-10-30 19:37:18.224276925 +0000 UTC m=+1820.221192310"
Oct 30 19:37:18.386296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:18.386209    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:18.386466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:18.386280    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:19.179533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:19.179490    6777 generic.go:356] "Generic (PLEG): container finished" podID="d03716bc-386f-4728-9b5a-7b28cf5b93d7" containerID="adfcc77d2983a298328761504c870d3f2749517f11d83aec75301e6a0522051b" exitCode=0
Oct 30 19:37:19.179952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:19.179633    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2" event={"ID":"d03716bc-386f-4728-9b5a-7b28cf5b93d7","Type":"ContainerDied","Data":"adfcc77d2983a298328761504c870d3f2749517f11d83aec75301e6a0522051b"}
Oct 30 19:37:20.213035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.213003    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"]
Oct 30 19:37:20.296073 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.296029    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"]
Oct 30 19:37:20.296256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.296167    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.428024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.427992    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc394be-a643-4647-9999-d163fc1a2fad-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.428241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.428049    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.428323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.428236    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.428323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.428290    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnjm\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-kube-api-access-trnjm\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.428420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.428330    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.428512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.428487    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.428558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.428530    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc394be-a643-4647-9999-d163fc1a2fad-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.428650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.428632    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.428776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.428762    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.447830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.447760    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:20.530459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.530423    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03716bc-386f-4728-9b5a-7b28cf5b93d7-kube-api-access-dzc8h\" (UniqueName: \"kubernetes.io/projected/d03716bc-386f-4728-9b5a-7b28cf5b93d7-kube-api-access-dzc8h\") pod \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") "
Oct 30 19:37:20.530664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.530507    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-xks-ca\") pod \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") "
Oct 30 19:37:20.530664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.530612    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-openshift-ca\") pod \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\" (UID: \"d03716bc-386f-4728-9b5a-7b28cf5b93d7\") "
Oct 30 19:37:20.530976 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.530937    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc394be-a643-4647-9999-d163fc1a2fad-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.531208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.531034    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.531338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.531318    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-openshift-ca" pod "d03716bc-386f-4728-9b5a-7b28cf5b93d7" (UID: "d03716bc-386f-4728-9b5a-7b28cf5b93d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:20.531652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.531562    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.531813    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnjm\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-kube-api-access-trnjm\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.531865    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.531988    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.532025    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc394be-a643-4647-9999-d163fc1a2fad-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.532096    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.532240    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.532263    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.532362    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc394be-a643-4647-9999-d163fc1a2fad-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.532514    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.532822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.532740    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.533568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.532900    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:20.533568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.532982    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.534137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.533996    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03716bc-386f-4728-9b5a-7b28cf5b93d7-kube-api-access-dzc8h" pod "d03716bc-386f-4728-9b5a-7b28cf5b93d7" (UID: "d03716bc-386f-4728-9b5a-7b28cf5b93d7"). InnerVolumeSpecName "kube-api-access-dzc8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:20.534629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.534606    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-xks-ca" pod "d03716bc-386f-4728-9b5a-7b28cf5b93d7" (UID: "d03716bc-386f-4728-9b5a-7b28cf5b93d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:20.535171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.535153    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.536367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.536350    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc394be-a643-4647-9999-d163fc1a2fad-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.544682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.544664    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.545238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.545214    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnjm\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-kube-api-access-trnjm\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:20.646731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.645596    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzc8h\" (UniqueName: \"kubernetes.io/projected/d03716bc-386f-4728-9b5a-7b28cf5b93d7-kube-api-access-dzc8h\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:20.646731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.645638    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03716bc-386f-4728-9b5a-7b28cf5b93d7-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:20.655761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:20.655429    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:21.000957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.000928    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"]
Oct 30 19:37:21.002623 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:37:21.002581    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc394be_a643_4647_9999_d163fc1a2fad.slice/crio-fccfc2abaeb13a03bc5e72e5d192eb00f42695a5bac14490bef6e70e9451faac WatchSource:0}: Error finding container fccfc2abaeb13a03bc5e72e5d192eb00f42695a5bac14490bef6e70e9451faac: Status 404 returned error can't find the container with id fccfc2abaeb13a03bc5e72e5d192eb00f42695a5bac14490bef6e70e9451faac
Oct 30 19:37:21.024581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.017704    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:21.024581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.017807    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:21.024581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.017862    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:21.202134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.202102    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"
Oct 30 19:37:21.202322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.202093    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2" event={"ID":"d03716bc-386f-4728-9b5a-7b28cf5b93d7","Type":"ContainerDied","Data":"56b2b438d118a5fac3143e1ff3804510b473458b4f48d1947264019376a8851c"}
Oct 30 19:37:21.202322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.202249    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56b2b438d118a5fac3143e1ff3804510b473458b4f48d1947264019376a8851c"
Oct 30 19:37:21.213345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.213315    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" event={"ID":"ffc394be-a643-4647-9999-d163fc1a2fad","Type":"ContainerStarted","Data":"b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec"}
Oct 30 19:37:21.213805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.213365    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" event={"ID":"ffc394be-a643-4647-9999-d163fc1a2fad","Type":"ContainerStarted","Data":"fccfc2abaeb13a03bc5e72e5d192eb00f42695a5bac14490bef6e70e9451faac"}
Oct 30 19:37:21.241340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.241277    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" podStartSLOduration=1.241263928 podStartE2EDuration="1.241263928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:21.239089399 +0000 UTC m=+1823.236004785" watchObservedRunningTime="2026-10-30 19:37:21.241263928 +0000 UTC m=+1823.238179311"
Oct 30 19:37:21.656395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:21.656362    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:22.662150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:22.658442    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:37:22.662150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:22.658509    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 19:37:23.225287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:23.225240    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:23.225475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:23.225303    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:23.672080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:23.671533    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:37:23.672080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:23.671596    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 19:37:24.545240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:24.545196    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:24.545424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:24.545265    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:24.684080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:24.673935    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:37:24.684080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:24.674871    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 19:37:25.676412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:25.676373    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:37:25.676828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:25.676798    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 19:37:26.674274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:26.673532    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:37:26.674274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:26.673605    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 19:37:26.890558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:26.890524    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"]
Oct 30 19:37:26.900548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:26.897257    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"]
Oct 30 19:37:27.730311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:27.730272    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:27.731129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:27.731102    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:27.732617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:27.732570    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:37:29.703228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.703187    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"]
Oct 30 19:37:29.704583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.704557    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d03716bc-386f-4728-9b5a-7b28cf5b93d7" containerName="revoke-keys"
Oct 30 19:37:29.704661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.704587    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03716bc-386f-4728-9b5a-7b28cf5b93d7" containerName="revoke-keys"
Oct 30 19:37:29.704893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.704878    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03716bc-386f-4728-9b5a-7b28cf5b93d7" containerName="revoke-keys"
Oct 30 19:37:29.905996 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.905952    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"]
Oct 30 19:37:29.906181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.906109    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:29.925898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.925857    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:29.926076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.925925    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:29.926150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.926047    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:29.926206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.926184    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:29.926370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.926345    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tj5\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-kube-api-access-h5tj5\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:29.926566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.926542    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:29.926668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.926647    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:29.926786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.926766    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:29.926822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:29.926799    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.027930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.027838    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.027930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.027888    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028007    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tj5\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-kube-api-access-h5tj5\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028178    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028263    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028347    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028373    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028466    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028495    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028509    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028639    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028720    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028771    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.028889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.028820    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.030416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.030397    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.030969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.030944    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.040868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.040842    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.040982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.040932    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tj5\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-kube-api-access-h5tj5\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.269778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.269746    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:30.878829 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:37:30.878799    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a53f7aa_0c6e_4f0c_8e86_8d20cf6a9ea4.slice/crio-6dfb0449c79f5552b0ddbb65609e128aafd9d38400407d36902fbbc01f87892e WatchSource:0}: Error finding container 6dfb0449c79f5552b0ddbb65609e128aafd9d38400407d36902fbbc01f87892e: Status 404 returned error can't find the container with id 6dfb0449c79f5552b0ddbb65609e128aafd9d38400407d36902fbbc01f87892e
Oct 30 19:37:30.879813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.879768    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"]
Oct 30 19:37:30.882464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.882431    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:30.882570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.882507    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:30.882570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:30.882543    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:31.395255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:31.395210    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" event={"ID":"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4","Type":"ContainerStarted","Data":"ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525"}
Oct 30 19:37:31.395445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:31.395286    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" event={"ID":"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4","Type":"ContainerStarted","Data":"6dfb0449c79f5552b0ddbb65609e128aafd9d38400407d36902fbbc01f87892e"}
Oct 30 19:37:31.891868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:31.891400    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr" podUID="94ff59f2-0974-4f2a-8bd6-b27052afca3d" containerName="payload-processing" containerID="cri-o://80ccaa320fdd167037b02b670177893cc91055fca19b9e375ff2601a45d15e2e" gracePeriod=25
Oct 30 19:37:31.897812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:31.897764    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v" podUID="d4c2cc99-d020-4163-bdbb-0ac595964973" containerName="payload-pre-processing" containerID="cri-o://0daad8f8cce151da002b86b50c73d8531b4351e0a37a5eeefa80d32497a14481" gracePeriod=25
Oct 30 19:37:32.434428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:32.434363    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" podStartSLOduration=3.434342289 podStartE2EDuration="3.434342289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:32.431706679 +0000 UTC m=+1834.428622068" watchObservedRunningTime="2026-10-30 19:37:32.434342289 +0000 UTC m=+1834.431257678"
Oct 30 19:37:33.273353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:33.273304    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:33.275765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:33.275731    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:37:33.275897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:33.275798    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 19:37:33.387267 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:33.387213    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:33.387444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:33.387290    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:33.493600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:33.492831    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:37:34.278939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.278467    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:37:34.278939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.278531    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 19:37:34.525094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.522166    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"]
Oct 30 19:37:34.578032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.572764    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"]
Oct 30 19:37:34.578032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.572897    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.677087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.676300    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.677087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.676462    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.677087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.676522    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwxr\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-kube-api-access-qcwxr\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.677087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.676558    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.677087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.676731    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ba1746-3bcc-4b27-b840-5515b4ba2950-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.677087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.676771    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.677087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.676831    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.677087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.676860    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.677087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.676985    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.779081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.778894    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.778990    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwxr\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-kube-api-access-qcwxr\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.780170    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.780369    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ba1746-3bcc-4b27-b840-5515b4ba2950-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.780420    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.780490    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.780516    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.780630    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.780698    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.781274    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.781355    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.781631    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.781896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.781885    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.783087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.782943    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ba1746-3bcc-4b27-b840-5515b4ba2950-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.785023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.784997    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.786317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.786292    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.796633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.796573    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.796787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.796645    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwxr\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-kube-api-access-qcwxr\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:34.941962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:34.941872    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:35.377965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.344422    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:37:35.377965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.344481    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 19:37:35.498535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.498496    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"]
Oct 30 19:37:35.566433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.566399    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.640761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.612778    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.640761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.612835    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.640761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.612974    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.640761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.613007    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4405b1c4-1a26-4592-a367-e763601cba0c-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.640761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.613108    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4405b1c4-1a26-4592-a367-e763601cba0c-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.656572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.656258    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dtt\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-kube-api-access-62dtt\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.656572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.656331    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.656572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.656486    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.656572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.656518    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.748665    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"]
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.762614    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.762734    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.762822    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.762858    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.763080    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.763124    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4405b1c4-1a26-4592-a367-e763601cba0c-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.763234    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4405b1c4-1a26-4592-a367-e763601cba0c-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.763399    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dtt\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-kube-api-access-62dtt\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.763451    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.773199    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.773499    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.773740    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.774639    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4405b1c4-1a26-4592-a367-e763601cba0c-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.774916    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.812265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.776120    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:35.827275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:35.822477    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4405b1c4-1a26-4592-a367-e763601cba0c-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:36.218661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.218577    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:36.272673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.272638    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:37:36.272949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.272915    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 19:37:36.273992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.273952    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:36.281117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.281083    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:36.281338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.281317    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:36.324007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.323972    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"]
Oct 30 19:37:36.341904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.328740    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dtt\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-kube-api-access-62dtt\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:36.433090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.431750    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:36.452088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.444760    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"]
Oct 30 19:37:36.470084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.466086    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.473122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.473073    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-8e6762\"/\"openshift-service-ca.crt\""
Oct 30 19:37:36.473289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.473165    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-8e6762\"/\"kube-root-ca.crt\""
Oct 30 19:37:36.473390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.473345    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-8e6762\"/\"default-dockercfg-2jbhp\""
Oct 30 19:37:36.473455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.473411    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-8e6762\"/\"auth-test-model-8e6762-kserve-self-signed-certs\""
Oct 30 19:37:36.528078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.527827    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"]
Oct 30 19:37:36.549111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.548161    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-home\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.549111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.548271    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-dshm\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.549111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.548403    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsq5d\" (UniqueName: \"kubernetes.io/projected/55866653-01c9-4307-9d74-cb117399e0d5-kube-api-access-bsq5d\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.549111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.548494    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-model-cache\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.549111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.548555    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-tmp-dir\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.549111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.548642    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55866653-01c9-4307-9d74-cb117399e0d5-tls-certs\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.675199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.672288    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-home\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.675199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.672405    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-dshm\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.675199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.672549    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsq5d\" (UniqueName: \"kubernetes.io/projected/55866653-01c9-4307-9d74-cb117399e0d5-kube-api-access-bsq5d\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.675199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.672645    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-model-cache\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.675199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.672710    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-tmp-dir\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.675199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.672808    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55866653-01c9-4307-9d74-cb117399e0d5-tls-certs\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.675199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.674836    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-home\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.684591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.684499    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55866653-01c9-4307-9d74-cb117399e0d5-tls-certs\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.686506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.684791    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-model-cache\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.686506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.685033    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-tmp-dir\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.714754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.713179    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-dshm\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.728618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.728551    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsq5d\" (UniqueName: \"kubernetes.io/projected/55866653-01c9-4307-9d74-cb117399e0d5-kube-api-access-bsq5d\") pod \"auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") " pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.823629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.823594    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" event={"ID":"79ba1746-3bcc-4b27-b840-5515b4ba2950","Type":"ContainerStarted","Data":"3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7"}
Oct 30 19:37:36.823912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.823869    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" event={"ID":"79ba1746-3bcc-4b27-b840-5515b4ba2950","Type":"ContainerStarted","Data":"ad6d754672480671fb6bd32b9f57236b3969ae543483ddc35b5ddfe381a7537b"}
Oct 30 19:37:36.881114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.881001    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:36.950165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.942376    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podStartSLOduration=2.942353249 podStartE2EDuration="2.942353249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:36.863614769 +0000 UTC m=+1838.860530158" watchObservedRunningTime="2026-10-30 19:37:36.942353249 +0000 UTC m=+1838.939268633"
Oct 30 19:37:36.950165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:36.943218    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:37.272592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:37.272206    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:37:37.272592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:37.272278    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 19:37:37.505145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:37.505111    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"]
Oct 30 19:37:37.556004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:37.555071    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:37.556004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:37.555186    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:37.556004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:37.555232    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:38.308083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:38.305610    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:37:38.308083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:38.305670    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 19:37:38.326159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:38.324377    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:38.326159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:38.324467    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:38.674090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:38.662898    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"]
Oct 30 19:37:38.791083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:38.744823    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" event={"ID":"4405b1c4-1a26-4592-a367-e763601cba0c","Type":"ContainerStarted","Data":"02c72e35df85fc1f0889c74625e41bd91f77967305aeaafff5e996e4f5c5e4f2"}
Oct 30 19:37:38.945989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:38.945952    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:37:38.946342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:38.946311    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 19:37:39.156228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.147910    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:39.156228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.149860    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:39.156228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.154241    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:37:39.567086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.560896    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:37:39.621712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.592203    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": context deadline exceeded"
Oct 30 19:37:39.624180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.622091    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": context deadline exceeded"
Oct 30 19:37:39.624180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.622566    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/ready\": context deadline exceeded"
Oct 30 19:37:39.624180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.622820    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:37:39.667108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.666384    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:39.667108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.666450    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:39.695372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:39.691338    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn" podUID="ba70864a-66bd-44b6-9923-0ac9f00260fd" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 30 19:37:40.032612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.024317    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:40.045084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.039475    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:37:40.082625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.039541    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 19:37:40.345076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.344840    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" event={"ID":"55866653-01c9-4307-9d74-cb117399e0d5","Type":"ContainerStarted","Data":"6ee1ba425c24a8f623140d1a86f2698a0f92407a8cefc01c9fd8f3bbee893ec5"}
Oct 30 19:37:40.374082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.373123    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" event={"ID":"4405b1c4-1a26-4592-a367-e763601cba0c","Type":"ContainerStarted","Data":"028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8"}
Oct 30 19:37:40.440979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.440948    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:40.441262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.441231    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:37:40.441343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.441292    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 19:37:40.707880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.707755    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" podStartSLOduration=5.707731741 podStartE2EDuration="5.707731741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:40.455787609 +0000 UTC m=+1842.452703036" watchObservedRunningTime="2026-10-30 19:37:40.707731741 +0000 UTC m=+1842.704647128"
Oct 30 19:37:40.966189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.965687    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:37:40.966189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:40.965758    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 19:37:41.535086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:41.464621    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:37:41.535086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:41.464696    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 19:37:41.689210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:41.689172    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" event={"ID":"55866653-01c9-4307-9d74-cb117399e0d5","Type":"ContainerStarted","Data":"a6f8c7c6783cba1587526b865245c4530665608b5cb9b4b4decf06c166bfb27a"}
Oct 30 19:37:41.737414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:41.737336    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podStartSLOduration=5.737315897 podStartE2EDuration="5.737315897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:41.729550414 +0000 UTC m=+1843.726465800" watchObservedRunningTime="2026-10-30 19:37:41.737315897 +0000 UTC m=+1843.734231286"
Oct 30 19:37:41.906506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:41.848720    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr" podUID="94ff59f2-0974-4f2a-8bd6-b27052afca3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:37:41.906506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:41.859496    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v" podUID="d4c2cc99-d020-4163-bdbb-0ac595964973" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:37:41.927588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:41.919551    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:41.971083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:41.948687    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:37:41.971083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:41.948775    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 19:37:42.443253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:42.439352    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:37:42.443253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:42.439416    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 19:37:42.944459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:42.944418    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:37:42.944858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:42.944485    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 19:37:43.455082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:43.444528    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:37:43.455082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:43.444590    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 19:37:43.602220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:43.602179    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"]
Oct 30 19:37:43.614267 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:43.614219    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs" podUID="527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" containerName="maas-api" containerID="cri-o://8e099b5430764ea3eae3fc9dcc66308aee0c6af7a7ce23b6a9cc6d2b7bba77ae" gracePeriod=30
Oct 30 19:37:44.000137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:43.976174    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:37:44.000137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:43.976255    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 19:37:44.323684 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:37:44.322450    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:37:44.332872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:44.332817    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"]
Oct 30 19:37:44.336964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:44.336940    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c24c01-sltk2"]
Oct 30 19:37:44.464085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:44.443085    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:37:44.464085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:44.443152    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 19:37:45.130982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.130606    6777 generic.go:356] "Generic (PLEG): container finished" podID="527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" containerID="8e099b5430764ea3eae3fc9dcc66308aee0c6af7a7ce23b6a9cc6d2b7bba77ae" exitCode=0
Oct 30 19:37:45.130982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.130859    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs" event={"ID":"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce","Type":"ContainerDied","Data":"8e099b5430764ea3eae3fc9dcc66308aee0c6af7a7ce23b6a9cc6d2b7bba77ae"}
Oct 30 19:37:45.130982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.130921    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs" event={"ID":"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce","Type":"ContainerDied","Data":"a8931be8414ddd5482a9473ebc575a557533a2de340376b792f30e98f8354866"}
Oct 30 19:37:45.130982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.130950    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8931be8414ddd5482a9473ebc575a557533a2de340376b792f30e98f8354866"
Oct 30 19:37:45.224372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.222669    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:45.224581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.224489    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:45.225554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.225528    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:37:45.278087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.275378    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:37:45.386484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.386381    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"]
Oct 30 19:37:45.387845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.387816    6777 cpu_manager.go:414] "containerMap: removing container" podUID="527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" containerName="maas-api"
Oct 30 19:37:45.387845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.387843    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" containerName="maas-api"
Oct 30 19:37:45.388077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.388051    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" containerName="maas-api"
Oct 30 19:37:45.399779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.399509    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-maas-api-tls\") pod \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") "
Oct 30 19:37:45.399779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.399636    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-openshift-service-ca\") pod \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") "
Oct 30 19:37:45.400561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.400110    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-kube-api-access-qbnr2\" (UniqueName: \"kubernetes.io/projected/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-kube-api-access-qbnr2\") pod \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") "
Oct 30 19:37:45.400561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.400254    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-metrics-certs\") pod \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\" (UID: \"527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce\") "
Oct 30 19:37:45.402618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.402584    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-openshift-service-ca" pod "527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" (UID: "527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:45.429094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.422521    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-metrics-certs" pod "527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" (UID: "527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:45.429094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.427847    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-kube-api-access-qbnr2" pod "527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" (UID: "527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce"). InnerVolumeSpecName "kube-api-access-qbnr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:45.436243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.434005    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-maas-api-tls" pod "527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" (UID: "527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:45.472590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.443262    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:37:45.472590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.443316    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 19:37:45.509094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.507673    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-maas-api-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:45.509094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.507710    6777 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-openshift-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:45.509094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.507730    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbnr2\" (UniqueName: \"kubernetes.io/projected/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-kube-api-access-qbnr2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:45.538321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.526443    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"]
Oct 30 19:37:45.538321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.526581    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.550989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.540277    6777 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce-metrics-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:45.641702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.641608    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.641702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.641659    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01352da7-5b23-44ec-addb-79a6f4a364b3-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.641934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.641735    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.641934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.641814    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.641934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.641923    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.642159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.642034    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.642159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.642120    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmkk\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-kube-api-access-xvmkk\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.642281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.642215    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.642339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.642297    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.744449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.744408    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.744625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.744524    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.744625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.744575    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.744625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.744605    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01352da7-5b23-44ec-addb-79a6f4a364b3-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.744757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.744681    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.744798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.744765    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.744902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.744885    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.745023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.745000    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.745023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.745005    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.745321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.745257    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmkk\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-kube-api-access-xvmkk\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.745391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.745362    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.745679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.745655    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.745787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.745692    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01352da7-5b23-44ec-addb-79a6f4a364b3-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.745863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.745804    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.747507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.747479    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.747961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.747939    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.762650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.762592    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.764917 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.764861    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmkk\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-kube-api-access-xvmkk\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:45.936572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:45.935959    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:46.153581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.153544    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"
Oct 30 19:37:46.161333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.161207    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03716bc-386f-4728-9b5a-7b28cf5b93d7" path="/var/lib/kubelet/pods/d03716bc-386f-4728-9b5a-7b28cf5b93d7/volumes"
Oct 30 19:37:46.202084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.201995    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"]
Oct 30 19:37:46.213487 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.213453    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c24c01-779664c894-lnfbs"]
Oct 30 19:37:46.537272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.535022    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:46.537272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.536940    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:46.543124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.540388    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"]
Oct 30 19:37:46.544557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.544485    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:37:46.545968 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:37:46.545939    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01352da7_5b23_44ec_addb_79a6f4a364b3.slice/crio-b6b33ba481ce8c59f31139363d5cf181909a6f97ebc6abf41ffe568bf51f3cc0 WatchSource:0}: Error finding container b6b33ba481ce8c59f31139363d5cf181909a6f97ebc6abf41ffe568bf51f3cc0: Status 404 returned error can't find the container with id b6b33ba481ce8c59f31139363d5cf181909a6f97ebc6abf41ffe568bf51f3cc0
Oct 30 19:37:46.549986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.549948    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:46.550212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.550037    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:46.550212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.550101    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:46.894506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.894412    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:46.908807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:46.908773    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:47.169550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:47.169513    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" event={"ID":"01352da7-5b23-44ec-addb-79a6f4a364b3","Type":"ContainerStarted","Data":"b6b33ba481ce8c59f31139363d5cf181909a6f97ebc6abf41ffe568bf51f3cc0"}
Oct 30 19:37:47.193527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:47.193496    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:37:48.117440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:48.117403    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce" path="/var/lib/kubelet/pods/527730ba-24e1-4d87-9ffa-e0fcfb5cd2ce/volumes"
Oct 30 19:37:48.183573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:48.183533    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" event={"ID":"01352da7-5b23-44ec-addb-79a6f4a364b3","Type":"ContainerStarted","Data":"dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d"}
Oct 30 19:37:48.214706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:48.214638    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podStartSLOduration=13.21462255 podStartE2EDuration="13.21462255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:48.209933988 +0000 UTC m=+1850.206849372" watchObservedRunningTime="2026-10-30 19:37:48.21462255 +0000 UTC m=+1850.211537932"
Oct 30 19:37:48.387299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:48.387209    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:48.387299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:48.387284    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:48.936501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:48.936459    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:50.003563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:48.938445    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:37:50.003563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:48.938500    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 30 19:37:50.003563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:49.576036    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:50.003563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:49.576377    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:50.003563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:49.936895    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:37:50.003563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:49.936968    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 30 19:37:50.296140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.278831    6777 generic.go:356] "Generic (PLEG): container finished" podID="132771fd-242f-43c0-9c51-32dc14c71f98" containerID="a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f" exitCode=0
Oct 30 19:37:50.296140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.279312    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" event={"ID":"132771fd-242f-43c0-9c51-32dc14c71f98","Type":"ContainerDied","Data":"a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f"}
Oct 30 19:37:50.298437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.296662    6777 generic.go:356] "Generic (PLEG): container finished" podID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerID="b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b" exitCode=0
Oct 30 19:37:50.298437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.296751    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" event={"ID":"1a01e0ca-b6ee-44c6-be57-f94a25d39f21","Type":"ContainerDied","Data":"b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b"}
Oct 30 19:37:50.314730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.314697    6777 scope.go:122] "RemoveContainer" containerID="a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f"
Oct 30 19:37:50.326101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.326046    6777 scope.go:122] "RemoveContainer" containerID="b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b"
Oct 30 19:37:50.332581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.332548    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:50.332685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.332616    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:50.332685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.332649    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:50.332800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.332691    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:50.332800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.332750    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:50.332800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.332783    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:50.936404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.936324    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:37:50.936404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:50.936389    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 30 19:37:51.345080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.333501    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" event={"ID":"1a01e0ca-b6ee-44c6-be57-f94a25d39f21","Type":"ContainerStarted","Data":"5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9"}
Oct 30 19:37:51.345080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.337390    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" event={"ID":"132771fd-242f-43c0-9c51-32dc14c71f98","Type":"ContainerStarted","Data":"392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d"}
Oct 30 19:37:51.624042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.623958    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:37:51.624904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.624879    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:51.625012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.624927    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 19:37:51.698887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.698856    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:37:51.700084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.700048    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:37:51.700230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.700102    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:37:51.828108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.827326    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr" podUID="94ff59f2-0974-4f2a-8bd6-b27052afca3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:37:51.843139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.837689    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v" podUID="d4c2cc99-d020-4163-bdbb-0ac595964973" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:37:51.954142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.942927    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:37:51.954142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:51.943033    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 30 19:37:52.325241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.322529    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:37:52.429084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.413445    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"]
Oct 30 19:37:52.429084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.413693    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy" containerID="cri-o://ae5c32ac92ee615b8d39509366f51e304d06eb2981e152c1240c53b2440757cb" gracePeriod=30
Oct 30 19:37:52.690181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.686565    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:52.690181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.686628    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 19:37:52.716448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.698382    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:37:52.716448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.698438    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:37:52.769077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.765217    6777 generic.go:356] "Generic (PLEG): container finished" podID="94ff59f2-0974-4f2a-8bd6-b27052afca3d" containerID="80ccaa320fdd167037b02b670177893cc91055fca19b9e375ff2601a45d15e2e" exitCode=0
Oct 30 19:37:52.769077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.765272    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr" event={"ID":"94ff59f2-0974-4f2a-8bd6-b27052afca3d","Type":"ContainerDied","Data":"80ccaa320fdd167037b02b670177893cc91055fca19b9e375ff2601a45d15e2e"}
Oct 30 19:37:52.787286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.778485    6777 generic.go:356] "Generic (PLEG): container finished" podID="d4c2cc99-d020-4163-bdbb-0ac595964973" containerID="0daad8f8cce151da002b86b50c73d8531b4351e0a37a5eeefa80d32497a14481" exitCode=0
Oct 30 19:37:52.787286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.779795    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v" event={"ID":"d4c2cc99-d020-4163-bdbb-0ac595964973","Type":"ContainerDied","Data":"0daad8f8cce151da002b86b50c73d8531b4351e0a37a5eeefa80d32497a14481"}
Oct 30 19:37:52.954527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.954393    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"]
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.979706    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:52.979787    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.100693    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.177198    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"]
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.178594    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.178698    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wsjq\" (UniqueName: \"kubernetes.io/projected/13d496cc-7e39-4fb9-bb18-d494e41e262e-kube-api-access-5wsjq\") pod \"maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.178820    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.213984    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.339324    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wsjq\" (UniqueName: \"kubernetes.io/projected/13d496cc-7e39-4fb9-bb18-d494e41e262e-kube-api-access-5wsjq\") pod \"maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.339508    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.340031    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.342129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.341081    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.428716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.428586    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:53.496955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.491190    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wsjq\" (UniqueName: \"kubernetes.io/projected/13d496cc-7e39-4fb9-bb18-d494e41e262e-kube-api-access-5wsjq\") pod \"maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.528919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.512291    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:37:53.528919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.513193    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:53.549605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.547733    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.625444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.625354    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:53.625444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.625429    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 19:37:53.650142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.647715    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:37:53.769824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.769790    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:37:53.770543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.770510    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:37:53.938786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.938204    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:37:53.938786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:53.938267    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 30 19:37:54.176652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.171663    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:37:54.322079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.319977    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c2cc99-d020-4163-bdbb-0ac595964973-kube-api-access-s59nh\" (UniqueName: \"kubernetes.io/projected/d4c2cc99-d020-4163-bdbb-0ac595964973-kube-api-access-s59nh\") pod \"d4c2cc99-d020-4163-bdbb-0ac595964973\" (UID: \"d4c2cc99-d020-4163-bdbb-0ac595964973\") "
Oct 30 19:37:54.322079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.320044    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c2cc99-d020-4163-bdbb-0ac595964973-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c2cc99-d020-4163-bdbb-0ac595964973-plugins-config-volume\") pod \"d4c2cc99-d020-4163-bdbb-0ac595964973\" (UID: \"d4c2cc99-d020-4163-bdbb-0ac595964973\") "
Oct 30 19:37:54.370137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.345295    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c2cc99-d020-4163-bdbb-0ac595964973-plugins-config-volume" pod "d4c2cc99-d020-4163-bdbb-0ac595964973" (UID: "d4c2cc99-d020-4163-bdbb-0ac595964973"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:54.423497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.423462    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c2cc99-d020-4163-bdbb-0ac595964973-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:54.483949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.463444    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c2cc99-d020-4163-bdbb-0ac595964973-kube-api-access-s59nh" pod "d4c2cc99-d020-4163-bdbb-0ac595964973" (UID: "d4c2cc99-d020-4163-bdbb-0ac595964973"). InnerVolumeSpecName "kube-api-access-s59nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:54.668275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.568242    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s59nh\" (UniqueName: \"kubernetes.io/projected/d4c2cc99-d020-4163-bdbb-0ac595964973-kube-api-access-s59nh\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:54.668275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.627769    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:54.668275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.627831    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 19:37:54.789231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.789186    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:37:54.789594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.789259    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:37:54.891900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.891863    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v" event={"ID":"d4c2cc99-d020-4163-bdbb-0ac595964973","Type":"ContainerDied","Data":"4d130c862e505c939fc42fa3af35e1227b2f81cda0c02a2665932d66d6887962"}
Oct 30 19:37:54.894394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.892256    6777 scope.go:122] "RemoveContainer" containerID="0daad8f8cce151da002b86b50c73d8531b4351e0a37a5eeefa80d32497a14481"
Oct 30 19:37:54.894394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.892497    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"
Oct 30 19:37:54.938252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.938231    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:37:54.938474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:54.938283    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 30 19:37:55.066087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.048565    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"]
Oct 30 19:37:55.123328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.123295    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d4c2cc99-d020-4163-bdbb-0ac595964973" containerName="payload-pre-processing"
Oct 30 19:37:55.123328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.123324    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c2cc99-d020-4163-bdbb-0ac595964973" containerName="payload-pre-processing"
Oct 30 19:37:55.123616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.123479    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c2cc99-d020-4163-bdbb-0ac595964973" containerName="payload-pre-processing"
Oct 30 19:37:55.182105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.165904    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:37:55.203148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.176376    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"]
Oct 30 19:37:55.229332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.225187    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"]
Oct 30 19:37:55.229332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.225236    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c24c01-6679b9f99-zf58v"]
Oct 30 19:37:55.229332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.178423    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.300081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.280808    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ff59f2-0974-4f2a-8bd6-b27052afca3d-kube-api-access-wqn4f\" (UniqueName: \"kubernetes.io/projected/94ff59f2-0974-4f2a-8bd6-b27052afca3d-kube-api-access-wqn4f\") pod \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\" (UID: \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\") "
Oct 30 19:37:55.300081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.281052    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ff59f2-0974-4f2a-8bd6-b27052afca3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ff59f2-0974-4f2a-8bd6-b27052afca3d-plugins-config-volume\") pod \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\" (UID: \"94ff59f2-0974-4f2a-8bd6-b27052afca3d\") "
Oct 30 19:37:55.300081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.294630    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ff59f2-0974-4f2a-8bd6-b27052afca3d-plugins-config-volume" pod "94ff59f2-0974-4f2a-8bd6-b27052afca3d" (UID: "94ff59f2-0974-4f2a-8bd6-b27052afca3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:55.313418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.313381    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ff59f2-0974-4f2a-8bd6-b27052afca3d-kube-api-access-wqn4f" pod "94ff59f2-0974-4f2a-8bd6-b27052afca3d" (UID: "94ff59f2-0974-4f2a-8bd6-b27052afca3d"). InnerVolumeSpecName "kube-api-access-wqn4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:55.327484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.327188    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.327484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.327239    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.327721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.327522    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.327721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.327581    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.327721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.327628    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qrl\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-kube-api-access-w6qrl\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.327918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.327897    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.328070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.328029    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.328311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.328288    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.328500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.328486    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.328876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.328861    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ff59f2-0974-4f2a-8bd6-b27052afca3d-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:55.328971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.328960    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqn4f\" (UniqueName: \"kubernetes.io/projected/94ff59f2-0974-4f2a-8bd6-b27052afca3d-kube-api-access-wqn4f\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.627215    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.627279    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.628981    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.629039    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.629110    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qrl\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-kube-api-access-w6qrl\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.629370    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.629477    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.629500    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.629569    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.629711    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.629739    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.630312    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.630553    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.630767    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.633016    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.633703    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.640624    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.643083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.642808    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.713523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.709176    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:37:55.713523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.709236    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:37:55.726094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.722721    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qrl\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-kube-api-access-w6qrl\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.733639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.733582    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.823609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.823563    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"]
Oct 30 19:37:55.823824 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:37:55.823583    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d496cc_7e39_4fb9_bb18_d494e41e262e.slice/crio-568e671c11a2062522c939ae874bedf4fee56b675fa75f207a052161918fc1fd WatchSource:0}: Error finding container 568e671c11a2062522c939ae874bedf4fee56b675fa75f207a052161918fc1fd: Status 404 returned error can't find the container with id 568e671c11a2062522c939ae874bedf4fee56b675fa75f207a052161918fc1fd
Oct 30 19:37:55.902166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.901983    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:55.916100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.916070    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"
Oct 30 19:37:55.916249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.916173    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr" event={"ID":"94ff59f2-0974-4f2a-8bd6-b27052afca3d","Type":"ContainerDied","Data":"c3a5595df8b4c8a10a94755c8455a981cfa2498cb6108d53d4862da997060c23"}
Oct 30 19:37:55.916292 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.916252    6777 scope.go:122] "RemoveContainer" containerID="80ccaa320fdd167037b02b670177893cc91055fca19b9e375ff2601a45d15e2e"
Oct 30 19:37:55.918685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.918665    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk" event={"ID":"13d496cc-7e39-4fb9-bb18-d494e41e262e","Type":"ContainerStarted","Data":"568e671c11a2062522c939ae874bedf4fee56b675fa75f207a052161918fc1fd"}
Oct 30 19:37:55.920871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.920849    6777 generic.go:356] "Generic (PLEG): container finished" podID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerID="0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5" exitCode=0
Oct 30 19:37:55.920984 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.920927    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" event={"ID":"ba8d5446-5b96-41f1-815e-88c0d09b24a0","Type":"ContainerDied","Data":"0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5"}
Oct 30 19:37:55.921467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.921444    6777 scope.go:122] "RemoveContainer" containerID="0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5"
Oct 30 19:37:55.923454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.923427    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:55.923537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.923493    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:55.923537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.923525    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:55.943053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.943028    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:55.943520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.943500    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:55.944401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:55.944382    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:37:56.421564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:56.421526    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c2cc99-d020-4163-bdbb-0ac595964973" path="/var/lib/kubelet/pods/d4c2cc99-d020-4163-bdbb-0ac595964973/volumes"
Oct 30 19:37:56.422110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:56.422085    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"]
Oct 30 19:37:56.422221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:56.422130    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c24c01-6b6df58646-phxcr"]
Oct 30 19:37:56.641734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:56.641261    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:56.641734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:56.641330    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 19:37:56.708100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:56.703173    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:37:56.708100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:56.703233    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:37:56.919801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:56.919712    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"]
Oct 30 19:37:57.026169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.026109    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:57.026169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.026209    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:57.026169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.026253    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:37:57.043181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.043131    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" event={"ID":"ba8d5446-5b96-41f1-815e-88c0d09b24a0","Type":"ContainerStarted","Data":"b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb"}
Oct 30 19:37:57.080082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.078693    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk" event={"ID":"13d496cc-7e39-4fb9-bb18-d494e41e262e","Type":"ContainerStarted","Data":"3a62f769d7539cc8f621ea00a90de9dff6e29cfa8178b89a284fbf0c7127e2cf"}
Oct 30 19:37:57.144091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.143860    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk" podStartSLOduration=5.143840727 podStartE2EDuration="5.143840727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:57.1416856 +0000 UTC m=+1859.138600984" watchObservedRunningTime="2026-10-30 19:37:57.143840727 +0000 UTC m=+1859.140756111"
Oct 30 19:37:57.633172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.630508    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:57.633172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.630567    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 19:37:57.715524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.713655    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:37:57.715524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.713716    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:37:57.867491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.866425    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:37:57.868312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.867901    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:57.868312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:57.867972    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 19:37:58.416934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.382127    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ff59f2-0974-4f2a-8bd6-b27052afca3d" path="/var/lib/kubelet/pods/94ff59f2-0974-4f2a-8bd6-b27052afca3d/volumes"
Oct 30 19:37:58.416934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.414329    6777 generic.go:356] "Generic (PLEG): container finished" podID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerID="ae5c32ac92ee615b8d39509366f51e304d06eb2981e152c1240c53b2440757cb" exitCode=0
Oct 30 19:37:58.419131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.419091    6777 generic.go:356] "Generic (PLEG): container finished" podID="13d496cc-7e39-4fb9-bb18-d494e41e262e" containerID="3a62f769d7539cc8f621ea00a90de9dff6e29cfa8178b89a284fbf0c7127e2cf" exitCode=0
Oct 30 19:37:58.432328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.431582    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" event={"ID":"cc320727-adce-4d5f-bab8-3b6806c955e0","Type":"ContainerDied","Data":"ae5c32ac92ee615b8d39509366f51e304d06eb2981e152c1240c53b2440757cb"}
Oct 30 19:37:58.432328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.431651    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk" event={"ID":"13d496cc-7e39-4fb9-bb18-d494e41e262e","Type":"ContainerDied","Data":"3a62f769d7539cc8f621ea00a90de9dff6e29cfa8178b89a284fbf0c7127e2cf"}
Oct 30 19:37:58.432328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.431682    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" event={"ID":"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce","Type":"ContainerStarted","Data":"10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe"}
Oct 30 19:37:58.432328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.431707    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" event={"ID":"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce","Type":"ContainerStarted","Data":"e496a72886c17afb2885e9c5ee6f0ccd1b47608cf35adc3a6178af01217f186d"}
Oct 30 19:37:58.432328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.431736    6777 scope.go:122] "RemoveContainer" containerID="1757d769dc961492da3aaeaa70127548e1e1e506bc4ddd83dd248895377fcdf0"
Oct 30 19:37:58.517531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.517313    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" podStartSLOduration=23.517293465 podStartE2EDuration="23.517293465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:58.492648079 +0000 UTC m=+1860.489563464" watchObservedRunningTime="2026-10-30 19:37:58.517293465 +0000 UTC m=+1860.514208851"
Oct 30 19:37:58.647444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.642530    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:37:58.669341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.669310    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:37:58.705832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.703530    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 19:37:58.705832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.703592    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 19:37:58.747937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.747831    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:37:58.875105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.866297    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:58.875105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.866378    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.904704    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.904829    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:58.904890    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.010333    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-token\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-token\") pod \"cc320727-adce-4d5f-bab8-3b6806c955e0\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") "
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.010456    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-envoy\") pod \"cc320727-adce-4d5f-bab8-3b6806c955e0\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") "
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.010563    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-podinfo\") pod \"cc320727-adce-4d5f-bab8-3b6806c955e0\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") "
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.010633    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-kube-api-access-4xjmp\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-kube-api-access-4xjmp\") pod \"cc320727-adce-4d5f-bab8-3b6806c955e0\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") "
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.010770    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc320727-adce-4d5f-bab8-3b6806c955e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc320727-adce-4d5f-bab8-3b6806c955e0-istiod-ca-cert\") pod \"cc320727-adce-4d5f-bab8-3b6806c955e0\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") "
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.010838    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-certs\") pod \"cc320727-adce-4d5f-bab8-3b6806c955e0\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") "
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.010866    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-socket\") pod \"cc320727-adce-4d5f-bab8-3b6806c955e0\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") "
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.010993    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-credential-socket\") pod \"cc320727-adce-4d5f-bab8-3b6806c955e0\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") "
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.011034    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-data\") pod \"cc320727-adce-4d5f-bab8-3b6806c955e0\" (UID: \"cc320727-adce-4d5f-bab8-3b6806c955e0\") "
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.018505    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-data" pod "cc320727-adce-4d5f-bab8-3b6806c955e0" (UID: "cc320727-adce-4d5f-bab8-3b6806c955e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.019080    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc320727-adce-4d5f-bab8-3b6806c955e0-istiod-ca-cert" pod "cc320727-adce-4d5f-bab8-3b6806c955e0" (UID: "cc320727-adce-4d5f-bab8-3b6806c955e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.019335    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-certs" pod "cc320727-adce-4d5f-bab8-3b6806c955e0" (UID: "cc320727-adce-4d5f-bab8-3b6806c955e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.019496    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-socket" pod "cc320727-adce-4d5f-bab8-3b6806c955e0" (UID: "cc320727-adce-4d5f-bab8-3b6806c955e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.019646    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-credential-socket" pod "cc320727-adce-4d5f-bab8-3b6806c955e0" (UID: "cc320727-adce-4d5f-bab8-3b6806c955e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.029666    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-podinfo" pod "cc320727-adce-4d5f-bab8-3b6806c955e0" (UID: "cc320727-adce-4d5f-bab8-3b6806c955e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.029862    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-token" pod "cc320727-adce-4d5f-bab8-3b6806c955e0" (UID: "cc320727-adce-4d5f-bab8-3b6806c955e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.042375    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-kube-api-access-4xjmp" pod "cc320727-adce-4d5f-bab8-3b6806c955e0" (UID: "cc320727-adce-4d5f-bab8-3b6806c955e0"). InnerVolumeSpecName "kube-api-access-4xjmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:59.043165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.042744    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-envoy" pod "cc320727-adce-4d5f-bab8-3b6806c955e0" (UID: "cc320727-adce-4d5f-bab8-3b6806c955e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:59.071629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.046218    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"]
Oct 30 19:37:59.111082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.108598    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy"
Oct 30 19:37:59.111082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.108642    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy"
Oct 30 19:37:59.111082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.108694    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy"
Oct 30 19:37:59.111082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.108717    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy"
Oct 30 19:37:59.111082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.108755    6777 cpu_manager.go:414] "containerMap: removing container" podUID="94ff59f2-0974-4f2a-8bd6-b27052afca3d" containerName="payload-processing"
Oct 30 19:37:59.111082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.108763    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ff59f2-0974-4f2a-8bd6-b27052afca3d" containerName="payload-processing"
Oct 30 19:37:59.111082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.109098    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy"
Oct 30 19:37:59.111082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.109118    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ff59f2-0974-4f2a-8bd6-b27052afca3d" containerName="payload-processing"
Oct 30 19:37:59.111082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.109159    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" containerName="istio-proxy"
Oct 30 19:37:59.114634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.113858    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.114634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.113886    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.114634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.113901    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.114634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.113916    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xjmp\" (UniqueName: \"kubernetes.io/projected/cc320727-adce-4d5f-bab8-3b6806c955e0-kube-api-access-4xjmp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.114634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.113930    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc320727-adce-4d5f-bab8-3b6806c955e0-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.114634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.113945    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.114634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.113959    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.114634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.113975    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.114634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.113988    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc320727-adce-4d5f-bab8-3b6806c955e0-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.139773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.139683    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"]
Oct 30 19:37:59.140100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.139795    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.215160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.215121    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.215160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.215164    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.215411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.215203    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.215411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.215276    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.215411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.215323    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.215411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.215343    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxqf\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-kube-api-access-hdxqf\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.215411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.215370    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.215694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.215463    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.215694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.215542    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.317654    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.317722    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.317797    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.317896    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.317976    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.318005    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxqf\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-kube-api-access-hdxqf\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.318048    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.318211    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.318304    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.319472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.319143    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.320181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.319762    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.320850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.320828    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.321025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.320901    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.321206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.321188    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.323491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.322798    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.325039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.324686    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.334238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.334166    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxqf\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-kube-api-access-hdxqf\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.336519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.336486    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.437498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.437344    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"
Oct 30 19:37:59.437498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.437365    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8" event={"ID":"cc320727-adce-4d5f-bab8-3b6806c955e0","Type":"ContainerDied","Data":"d3411d8a97ea31b04360a7b275f52a866bcac16797698dc90d10cc7b5a60723d"}
Oct 30 19:37:59.437498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.437452    6777 scope.go:122] "RemoveContainer" containerID="ae5c32ac92ee615b8d39509366f51e304d06eb2981e152c1240c53b2440757cb"
Oct 30 19:37:59.530094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.529912    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:37:59.531515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.531455    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"]
Oct 30 19:37:59.536509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.536483    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-pksh8"]
Oct 30 19:37:59.729219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.722563    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:37:59.729219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.725380    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:37:59.844115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.843972    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:59.844115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.844034    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 19:37:59.930088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.928958    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:37:59.930088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.929012    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 19:37:59.943200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:37:59.943166    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:38:00.117868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.117458    6777 scope.go:122] "RemoveContainer" containerID="26c846b3c6568675dced39f56dd25fbbd24cd4ed0444ac66515be5b86cfddf7a"
Oct 30 19:38:00.142168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.141948    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d496cc-7e39-4fb9-bb18-d494e41e262e-kube-api-access-5wsjq\" (UniqueName: \"kubernetes.io/projected/13d496cc-7e39-4fb9-bb18-d494e41e262e-kube-api-access-5wsjq\") pod \"13d496cc-7e39-4fb9-bb18-d494e41e262e\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") "
Oct 30 19:38:00.144032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.142898    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-xks-ca\") pod \"13d496cc-7e39-4fb9-bb18-d494e41e262e\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") "
Oct 30 19:38:00.144032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.143353    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-openshift-ca\") pod \"13d496cc-7e39-4fb9-bb18-d494e41e262e\" (UID: \"13d496cc-7e39-4fb9-bb18-d494e41e262e\") "
Oct 30 19:38:00.144245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.144136    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-openshift-ca" pod "13d496cc-7e39-4fb9-bb18-d494e41e262e" (UID: "13d496cc-7e39-4fb9-bb18-d494e41e262e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:00.147853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.145806    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.157098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.152496    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-xks-ca" pod "13d496cc-7e39-4fb9-bb18-d494e41e262e" (UID: "13d496cc-7e39-4fb9-bb18-d494e41e262e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:00.206289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.206255    6777 scope.go:122] "RemoveContainer" containerID="8e099b5430764ea3eae3fc9dcc66308aee0c6af7a7ce23b6a9cc6d2b7bba77ae"
Oct 30 19:38:00.206289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.206275    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d496cc-7e39-4fb9-bb18-d494e41e262e-kube-api-access-5wsjq" pod "13d496cc-7e39-4fb9-bb18-d494e41e262e" (UID: "13d496cc-7e39-4fb9-bb18-d494e41e262e"). InnerVolumeSpecName "kube-api-access-5wsjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:00.213777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.213710    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc320727-adce-4d5f-bab8-3b6806c955e0" path="/var/lib/kubelet/pods/cc320727-adce-4d5f-bab8-3b6806c955e0/volumes"
Oct 30 19:38:00.240278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.240249    6777 scope.go:122] "RemoveContainer" containerID="89130c7827dfef7baa82e00ebad80c4a7ca7efe4dfc0771530e56b03a7b9ad77"
Oct 30 19:38:00.277083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.266637    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d496cc-7e39-4fb9-bb18-d494e41e262e-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.277083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.266676    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wsjq\" (UniqueName: \"kubernetes.io/projected/13d496cc-7e39-4fb9-bb18-d494e41e262e-kube-api-access-5wsjq\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.314069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.314027    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"]
Oct 30 19:38:00.315014 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:38:00.314984    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd96ad7_71c6_4389_bf36_dca4475cb5e5.slice/crio-ffa05dcb1189fc68be02facd950f56a7c80cc252bc8ccfd10342be2f6ea4f5a3 WatchSource:0}: Error finding container ffa05dcb1189fc68be02facd950f56a7c80cc252bc8ccfd10342be2f6ea4f5a3: Status 404 returned error can't find the container with id ffa05dcb1189fc68be02facd950f56a7c80cc252bc8ccfd10342be2f6ea4f5a3
Oct 30 19:38:00.318877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.318843    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:00.319020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.318927    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:00.319020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.318980    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:00.621301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.621270    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" event={"ID":"2dd96ad7-71c6-4389-bf36-dca4475cb5e5","Type":"ContainerStarted","Data":"ffa05dcb1189fc68be02facd950f56a7c80cc252bc8ccfd10342be2f6ea4f5a3"}
Oct 30 19:38:00.624683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.624656    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk" event={"ID":"13d496cc-7e39-4fb9-bb18-d494e41e262e","Type":"ContainerDied","Data":"568e671c11a2062522c939ae874bedf4fee56b675fa75f207a052161918fc1fd"}
Oct 30 19:38:00.624773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.624708    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="568e671c11a2062522c939ae874bedf4fee56b675fa75f207a052161918fc1fd"
Oct 30 19:38:00.624808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.624784    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"
Oct 30 19:38:00.832743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.832710    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:38:00.832916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.832761    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 19:38:00.902629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.902509    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:38:00.902629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:00.902573    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 19:38:01.644424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:01.644391    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" event={"ID":"2dd96ad7-71c6-4389-bf36-dca4475cb5e5","Type":"ContainerStarted","Data":"a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee"}
Oct 30 19:38:01.833184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:01.833102    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:38:01.833184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:01.833167    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 19:38:01.907170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:01.903500    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:38:01.907170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:01.903566    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 19:38:02.344963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:02.344931    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:38:02.531484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:02.531449    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:38:02.533561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:02.533536    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:38:02.533649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:02.533589    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 19:38:02.844089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:02.842884    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:38:02.844089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:02.842946    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 19:38:02.911551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:02.911510    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:38:02.911886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:02.911853    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 19:38:03.010078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:03.008406    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" podStartSLOduration=14.008384889 podStartE2EDuration="14.008384889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:01.702126375 +0000 UTC m=+1863.699041762" watchObservedRunningTime="2026-10-30 19:38:03.008384889 +0000 UTC m=+1865.005300284"
Oct 30 19:38:03.056040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:03.048686    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"]
Oct 30 19:38:03.187369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:03.152111    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"]
Oct 30 19:38:03.698088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:03.688051    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:38:03.698088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:03.688128    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 19:38:03.937584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:03.937547    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:38:03.938081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:03.937619    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 19:38:03.939590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:03.939556    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:38:04.045207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:04.045169    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 19:38:04.531337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:04.531293    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:38:04.531538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:04.531371    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 19:38:04.626205 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:04.626161    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:38:04.626387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:04.626217    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:04.840102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:04.840002    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:38:04.842020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:04.841998    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:38:05.033215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.031961    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:38:05.033215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.032509    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:38:05.100209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.096470    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:38:05.113792 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.113756    6777 generic.go:356] "Generic (PLEG): container finished" podID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerID="f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb" exitCode=0
Oct 30 19:38:05.114638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.114352    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" event={"ID":"b316199b-90ea-49c2-92f9-31a270b16d4a","Type":"ContainerDied","Data":"f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb"}
Oct 30 19:38:05.114757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.114732    6777 scope.go:122] "RemoveContainer" containerID="f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb"
Oct 30 19:38:05.116919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.116886    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:05.117003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.116977    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:05.117041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.117023    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:05.531531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.531499    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:38:05.531732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:05.531551    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 19:38:06.124630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:06.124595    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" event={"ID":"b316199b-90ea-49c2-92f9-31a270b16d4a","Type":"ContainerStarted","Data":"adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386"}
Oct 30 19:38:06.531198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:06.531166    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:38:06.531377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:06.531227    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 19:38:06.922157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:06.922051    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:38:06.922315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:06.922164    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:38:06.922315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:06.922197    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 19:38:07.531539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:07.531501    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:38:07.531946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:07.531550    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 19:38:07.930127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:07.923434    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:38:07.930127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:07.923502    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 19:38:08.049272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.049184    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5" podUID="31a6c3c2-002f-45db-a84b-31bc40cb6dd5" containerName="payload-processing" containerID="cri-o://2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032" gracePeriod=25
Oct 30 19:38:08.163088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.153467    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd" podUID="f78e5730-0b9f-45b1-8e5a-16f1d96e3228" containerName="payload-pre-processing" containerID="cri-o://066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497" gracePeriod=25
Oct 30 19:38:08.428755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.428725    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:38:08.471132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.461947    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-plugins-config-volume\") pod \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\" (UID: \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\") "
Oct 30 19:38:08.471132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.462190    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-kube-api-access-wwcww\" (UniqueName: \"kubernetes.io/projected/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-kube-api-access-wwcww\") pod \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\" (UID: \"31a6c3c2-002f-45db-a84b-31bc40cb6dd5\") "
Oct 30 19:38:08.479086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.475493    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-plugins-config-volume" pod "31a6c3c2-002f-45db-a84b-31bc40cb6dd5" (UID: "31a6c3c2-002f-45db-a84b-31bc40cb6dd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:08.488723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.488661    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-kube-api-access-wwcww" pod "31a6c3c2-002f-45db-a84b-31bc40cb6dd5" (UID: "31a6c3c2-002f-45db-a84b-31bc40cb6dd5"). InnerVolumeSpecName "kube-api-access-wwcww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:08.491882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.491856    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"]
Oct 30 19:38:08.493199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.493174    6777 cpu_manager.go:414] "containerMap: removing container" podUID="31a6c3c2-002f-45db-a84b-31bc40cb6dd5" containerName="payload-processing"
Oct 30 19:38:08.493199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.493199    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a6c3c2-002f-45db-a84b-31bc40cb6dd5" containerName="payload-processing"
Oct 30 19:38:08.493364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.493284    6777 cpu_manager.go:414] "containerMap: removing container" podUID="13d496cc-7e39-4fb9-bb18-d494e41e262e" containerName="revoke-keys"
Oct 30 19:38:08.493364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.493294    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d496cc-7e39-4fb9-bb18-d494e41e262e" containerName="revoke-keys"
Oct 30 19:38:08.493526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.493507    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a6c3c2-002f-45db-a84b-31bc40cb6dd5" containerName="payload-processing"
Oct 30 19:38:08.493580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.493540    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d496cc-7e39-4fb9-bb18-d494e41e262e" containerName="revoke-keys"
Oct 30 19:38:08.499079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.499048    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.502455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.502427    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"]
Oct 30 19:38:08.503691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.503667    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ddb6ea-openshift-default-dockercfg-z5667\""
Oct 30 19:38:08.536207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.536181    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:38:08.537449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.537427    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:38:08.538027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.538007    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:38:08.576185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.575780    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-podinfo\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.576185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.575847    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-token\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.576185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.575932    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-envoy\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.576185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.576024    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.576185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.576081    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07861c1-e442-495a-9e32-231b53ae1c5d-istiod-ca-cert\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.576185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.576144    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7vq5\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-kube-api-access-v7vq5\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.576515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.576403    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-certs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.577035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.576597    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-data\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.577035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.576651    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-credential-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.577035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.576903    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:08.577035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.577033    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwcww\" (UniqueName: \"kubernetes.io/projected/31a6c3c2-002f-45db-a84b-31bc40cb6dd5-kube-api-access-wwcww\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:08.681426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.681390    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-data\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.681638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.681442    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-credential-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.681638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.681551    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-podinfo\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.681638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.681594    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-token\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.681835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.681678    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-envoy\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.681835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.681769    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.681835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.681813    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07861c1-e442-495a-9e32-231b53ae1c5d-istiod-ca-cert\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.682005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.681877    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7vq5\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-kube-api-access-v7vq5\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.682092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.682011    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-certs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.683258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.682558    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-certs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.683258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.682768    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-data\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.683258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.683051    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-credential-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.683669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.683639    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07861c1-e442-495a-9e32-231b53ae1c5d-istiod-ca-cert\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.684182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.684157    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.685466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.685037    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-envoy\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.685596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.685578    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-podinfo\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.697275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.697253    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-token\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.700151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.700103    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7vq5\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-kube-api-access-v7vq5\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.834091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.832426    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:08.921604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.921571    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:38:08.921744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:08.921630    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 19:38:09.180336 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:38:09.180303    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07861c1_e442_495a_9e32_231b53ae1c5d.slice/crio-6a45b23b8e30679e1c5bbc95a4560bac873e489217be84d30409984326bd320d WatchSource:0}: Error finding container 6a45b23b8e30679e1c5bbc95a4560bac873e489217be84d30409984326bd320d: Status 404 returned error can't find the container with id 6a45b23b8e30679e1c5bbc95a4560bac873e489217be84d30409984326bd320d
Oct 30 19:38:09.180487 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.180439    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"]
Oct 30 19:38:09.184461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.184425    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:09.184567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.184504    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:09.184567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.184546    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:09.185604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.185580    6777 generic.go:356] "Generic (PLEG): container finished" podID="31a6c3c2-002f-45db-a84b-31bc40cb6dd5" containerID="2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032" exitCode=0
Oct 30 19:38:09.185702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.185626    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"
Oct 30 19:38:09.185702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.185661    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5" event={"ID":"31a6c3c2-002f-45db-a84b-31bc40cb6dd5","Type":"ContainerDied","Data":"2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032"}
Oct 30 19:38:09.185778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.185718    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5" event={"ID":"31a6c3c2-002f-45db-a84b-31bc40cb6dd5","Type":"ContainerDied","Data":"35bb4fc092e455dde096acb6efd404bfe95c4641a199b2cbe6401d6cbcd3fdd2"}
Oct 30 19:38:09.185778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.185749    6777 scope.go:122] "RemoveContainer" containerID="2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032"
Oct 30 19:38:09.204161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.204144    6777 scope.go:122] "RemoveContainer" containerID="2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032"
Oct 30 19:38:09.204424 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:38:09.204404    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032\": container with ID starting with 2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032 not found: ID does not exist" containerID="2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032"
Oct 30 19:38:09.204480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.204431    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032"} err="failed to get container status \"2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032\": rpc error: code = NotFound desc = could not find container \"2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032\": container with ID starting with 2dd71f100a57e6fa9c377a27cac1cd37dd49f1312d15349c6bb46a0bfcd13032 not found: ID does not exist"
Oct 30 19:38:09.217359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.217336    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"]
Oct 30 19:38:09.222089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.222049    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-76117c-7b6dc4bd8-tjgr5"]
Oct 30 19:38:09.921569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.921531    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:38:09.922117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:09.921582    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 19:38:10.143135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:10.142119    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a6c3c2-002f-45db-a84b-31bc40cb6dd5" path="/var/lib/kubelet/pods/31a6c3c2-002f-45db-a84b-31bc40cb6dd5/volumes"
Oct 30 19:38:10.199898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:10.199801    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" event={"ID":"c07861c1-e442-495a-9e32-231b53ae1c5d","Type":"ContainerStarted","Data":"5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a"}
Oct 30 19:38:10.201033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:10.201002    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" event={"ID":"c07861c1-e442-495a-9e32-231b53ae1c5d","Type":"ContainerStarted","Data":"6a45b23b8e30679e1c5bbc95a4560bac873e489217be84d30409984326bd320d"}
Oct 30 19:38:10.231921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:10.231856    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" podStartSLOduration=11.23183918 podStartE2EDuration="11.23183918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:10.22463119 +0000 UTC m=+1872.221546573" watchObservedRunningTime="2026-10-30 19:38:10.23183918 +0000 UTC m=+1872.228754565"
Oct 30 19:38:10.835081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:10.833417    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:10.836630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:10.836591    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 19:38:10.836630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:10.836650    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 19:38:10.921885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:10.921855    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:38:10.922371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:10.921907    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 19:38:11.497081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:11.496095    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"]
Oct 30 19:38:11.545082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:11.544510    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76117c-7lhrk"]
Oct 30 19:38:11.844092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:11.835139    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 19:38:11.844092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:11.835201    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 19:38:11.924410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:11.924375    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:38:11.924867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:11.924445    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 19:38:12.141273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:12.140953    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d496cc-7e39-4fb9-bb18-d494e41e262e" path="/var/lib/kubelet/pods/13d496cc-7e39-4fb9-bb18-d494e41e262e/volumes"
Oct 30 19:38:12.141511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:12.141487    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"]
Oct 30 19:38:12.141606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:12.141534    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43e37f7-68mtl"]
Oct 30 19:38:12.833869 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:12.833814    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 19:38:12.834120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:12.833879    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 19:38:12.987354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:12.983445    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:38:12.987354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:12.984889    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:38:12.990098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:12.990074    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:38:13.695151 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:38:13.695116    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:38:13.834050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:13.834016    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 19:38:13.834264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:13.834110    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 19:38:14.117164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:14.117128    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f682adc-fe85-4781-9b5d-f0a9f9f6ba10" path="/var/lib/kubelet/pods/7f682adc-fe85-4781-9b5d-f0a9f9f6ba10/volumes"
Oct 30 19:38:14.833809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:14.833761    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 19:38:14.833982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:14.833826    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 19:38:15.773908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:15.773862    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd" podUID="f78e5730-0b9f-45b1-8e5a-16f1d96e3228" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:38:15.834097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:15.834024    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 19:38:15.834274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:15.834114    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 19:38:16.833724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:16.833685    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 19:38:16.834137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:16.833752    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 19:38:17.838504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:17.838472    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:17.839013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:17.838991    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:17.839930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:17.839902    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:38:19.775200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:19.775162    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"]
Oct 30 19:38:19.775693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:19.775498    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy" containerID="cri-o://33b672524a8ddddfaa5ffb07dc63af46321942c048eaca53e61465008f3a472b" gracePeriod=30
Oct 30 19:38:23.578097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:23.576180    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.581917 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:23.581788    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.713085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:23.712201    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.740094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:23.739193    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.917147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:23.912195    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.948541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:23.948496    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.949416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:23.949384    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:38:24.020087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.009351    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.065087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.046887    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.090154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.085339    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.179258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.178082    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.364165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.364120    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.431182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.425405    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.431980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.431936    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/ready\": context deadline exceeded"
Oct 30 19:38:24.472126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.464908    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.472126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.466126    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.591383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.591338    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": context deadline exceeded"
Oct 30 19:38:24.592804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.592776    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:38:24.784146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.777153    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": context deadline exceeded"
Oct 30 19:38:24.784146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.777629    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:38:24.784146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.778046    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:38:24.784146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:24.778132    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:38:25.023091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.013177    6777 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:38:25.023091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.013254    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.057141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.046004    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.057141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.047317    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:38:25.099082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.091140    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.099082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.091748    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:38:25.099082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.092244    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.099082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.093178    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.099082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.093700    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:38:25.099082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.093753    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:38:25.157418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.108336    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.157418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.110021    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.157418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.110645    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:38:25.157418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.110699    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:38:25.282084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.217141    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.282084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.254249    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:38:25.325150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.318137    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:38:25.325150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.318218    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.600145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.435973    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.600145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.436460    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:38:25.600145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.437689    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.600145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.438249    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:38:25.600145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.481862    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.600145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.482196    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.600145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.482770    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:38:25.600145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.482830    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:38:25.705145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.641464    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.729088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.724186    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:38:25.805905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.778513    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.805905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.778725    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.805905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.798982    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd" podUID="f78e5730-0b9f-45b1-8e5a-16f1d96e3228" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:38:25.850379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.845769    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:38:25.941179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.941150    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:38:25.994612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.990792    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:38:25.994855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:25.994831    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:38:26.041954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.041725    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:38:26.061361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.051836    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:38:26.061361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.051904    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:38:26.061361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.052005    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:38:26.061361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.052694    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:38:26.065932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.065595    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:38:26.096826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.090173    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:38:26.096826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.095142    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:38:26.147146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.142824    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:38:26.218737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.218270    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:38:26.226727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.226668    6777 generic.go:356] "Generic (PLEG): container finished" podID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerID="33b672524a8ddddfaa5ffb07dc63af46321942c048eaca53e61465008f3a472b" exitCode=0
Oct 30 19:38:26.230523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.229265    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" event={"ID":"a15081ca-227c-428f-ad9e-a5bac902df6e","Type":"ContainerDied","Data":"33b672524a8ddddfaa5ffb07dc63af46321942c048eaca53e61465008f3a472b"}
Oct 30 19:38:26.230523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.229405    6777 scope.go:122] "RemoveContainer" containerID="4b7738d58fe2859a719d9855578fbd91896fe6bba476c7b9f1fa63580a81f858"
Oct 30 19:38:26.450167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.448976    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:38:26.463557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.463517    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-socket\") pod \"a15081ca-227c-428f-ad9e-a5bac902df6e\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") "
Oct 30 19:38:26.463777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.463581    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-envoy\") pod \"a15081ca-227c-428f-ad9e-a5bac902df6e\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") "
Oct 30 19:38:26.463777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.463614    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-data\") pod \"a15081ca-227c-428f-ad9e-a5bac902df6e\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") "
Oct 30 19:38:26.463916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.463807    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-kube-api-access-cv59z\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-kube-api-access-cv59z\") pod \"a15081ca-227c-428f-ad9e-a5bac902df6e\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") "
Oct 30 19:38:26.463916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.463847    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-token\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-token\") pod \"a15081ca-227c-428f-ad9e-a5bac902df6e\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") "
Oct 30 19:38:26.463916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.463855    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-socket" pod "a15081ca-227c-428f-ad9e-a5bac902df6e" (UID: "a15081ca-227c-428f-ad9e-a5bac902df6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.463916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.463878    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-podinfo\") pod \"a15081ca-227c-428f-ad9e-a5bac902df6e\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") "
Oct 30 19:38:26.464144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.463984    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15081ca-227c-428f-ad9e-a5bac902df6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15081ca-227c-428f-ad9e-a5bac902df6e-istiod-ca-cert\") pod \"a15081ca-227c-428f-ad9e-a5bac902df6e\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") "
Oct 30 19:38:26.464144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.464023    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-credential-socket\") pod \"a15081ca-227c-428f-ad9e-a5bac902df6e\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") "
Oct 30 19:38:26.464265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.464189    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-certs\") pod \"a15081ca-227c-428f-ad9e-a5bac902df6e\" (UID: \"a15081ca-227c-428f-ad9e-a5bac902df6e\") "
Oct 30 19:38:26.464454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.464429    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15081ca-227c-428f-ad9e-a5bac902df6e-istiod-ca-cert" pod "a15081ca-227c-428f-ad9e-a5bac902df6e" (UID: "a15081ca-227c-428f-ad9e-a5bac902df6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:26.465404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.465381    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15081ca-227c-428f-ad9e-a5bac902df6e-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.465574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.465558    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.465680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.465524    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-data" pod "a15081ca-227c-428f-ad9e-a5bac902df6e" (UID: "a15081ca-227c-428f-ad9e-a5bac902df6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.465956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.465936    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-certs" pod "a15081ca-227c-428f-ad9e-a5bac902df6e" (UID: "a15081ca-227c-428f-ad9e-a5bac902df6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.466846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.466704    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-credential-socket" pod "a15081ca-227c-428f-ad9e-a5bac902df6e" (UID: "a15081ca-227c-428f-ad9e-a5bac902df6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.470491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.470467    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-token" pod "a15081ca-227c-428f-ad9e-a5bac902df6e" (UID: "a15081ca-227c-428f-ad9e-a5bac902df6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:26.478914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.478864    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-envoy" pod "a15081ca-227c-428f-ad9e-a5bac902df6e" (UID: "a15081ca-227c-428f-ad9e-a5bac902df6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.479675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.479636    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-podinfo" pod "a15081ca-227c-428f-ad9e-a5bac902df6e" (UID: "a15081ca-227c-428f-ad9e-a5bac902df6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:26.482484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.481307    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-kube-api-access-cv59z" pod "a15081ca-227c-428f-ad9e-a5bac902df6e" (UID: "a15081ca-227c-428f-ad9e-a5bac902df6e"). InnerVolumeSpecName "kube-api-access-cv59z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:26.583728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.581776    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.583728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.581813    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.583728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.581830    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.583728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.581845    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.583728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.581858    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv59z\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-kube-api-access-cv59z\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.583728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.581873    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.583728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.581886    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15081ca-227c-428f-ad9e-a5bac902df6e-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.591187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.590738    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"]
Oct 30 19:38:26.592084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.591984    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy"
Oct 30 19:38:26.592084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.592012    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy"
Oct 30 19:38:26.592084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.592076    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy"
Oct 30 19:38:26.592084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.592084    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy"
Oct 30 19:38:26.592448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.592290    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy"
Oct 30 19:38:26.620336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.620297    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"]
Oct 30 19:38:26.620809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.620445    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.628107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.627834    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-178bcc-openshift-default-dockercfg-6cflr\""
Oct 30 19:38:26.685578 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.685540    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3806290b-05d3-4460-8b13-eeef5668d2e4-istiod-ca-cert\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.685900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.685878    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-certs\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.686090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.686074    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-socket\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.686232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.686216    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-envoy\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.686354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.686340    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-token\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.686568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.686548    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82v7s\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-kube-api-access-82v7s\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.686737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.686724    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-credential-socket\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.686861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.686849    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-data\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.687019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.686999    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-podinfo\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.788440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788395    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-certs\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.788672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788459    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-socket\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.788672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788511    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-envoy\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.788672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788530    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-token\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.788672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788642    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82v7s\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-kube-api-access-82v7s\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.788920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788730    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-credential-socket\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.788920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788788    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-data\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.788920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788859    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-podinfo\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.789181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788925    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-certs\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.789181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788961    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-socket\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.789181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.788998    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3806290b-05d3-4460-8b13-eeef5668d2e4-istiod-ca-cert\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.789337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.789215    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-credential-socket\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.789337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.789258    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-data\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.789722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.789701    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3806290b-05d3-4460-8b13-eeef5668d2e4-istiod-ca-cert\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.791867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.791837    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-envoy\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.792216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.792185    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-podinfo\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.801770 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.801734    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-token\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:26.802658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:26.802631    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82v7s\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-kube-api-access-82v7s\") pod \"e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") " pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:27.027290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.026995    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:27.239526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.239497    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"
Oct 30 19:38:27.239705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.239533    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w" event={"ID":"a15081ca-227c-428f-ad9e-a5bac902df6e","Type":"ContainerDied","Data":"4bc83f4495c86339b98129f9b90c56c9d6a9c3fb674417c157e1f57c85005530"}
Oct 30 19:38:27.239705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.239587    6777 scope.go:122] "RemoveContainer" containerID="33b672524a8ddddfaa5ffb07dc63af46321942c048eaca53e61465008f3a472b"
Oct 30 19:38:27.268247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.268211    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"]
Oct 30 19:38:27.275574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.275539    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-68b4w"]
Oct 30 19:38:27.350888 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:38:27.350862    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3806290b_05d3_4460_8b13_eeef5668d2e4.slice/crio-0bc0d4f048c803cfbeec5ec3747c97e1e20e52d45f0041ee5293c2864ff4ac66 WatchSource:0}: Error finding container 0bc0d4f048c803cfbeec5ec3747c97e1e20e52d45f0041ee5293c2864ff4ac66: Status 404 returned error can't find the container with id 0bc0d4f048c803cfbeec5ec3747c97e1e20e52d45f0041ee5293c2864ff4ac66
Oct 30 19:38:27.351552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.351535    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"]
Oct 30 19:38:27.354597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.354557    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:27.354670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.354628    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:27.354670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:27.354657    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:38:28.120393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.120350    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" path="/var/lib/kubelet/pods/a15081ca-227c-428f-ad9e-a5bac902df6e/volumes"
Oct 30 19:38:28.255622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.255572    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" event={"ID":"3806290b-05d3-4460-8b13-eeef5668d2e4","Type":"ContainerStarted","Data":"d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce"}
Oct 30 19:38:28.255821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.255671    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" event={"ID":"3806290b-05d3-4460-8b13-eeef5668d2e4","Type":"ContainerStarted","Data":"0bc0d4f048c803cfbeec5ec3747c97e1e20e52d45f0041ee5293c2864ff4ac66"}
Oct 30 19:38:28.287349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.287287    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podStartSLOduration=7.287271148 podStartE2EDuration="7.287271148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:28.282866807 +0000 UTC m=+1890.279782192" watchObservedRunningTime="2026-10-30 19:38:28.287271148 +0000 UTC m=+1890.284186529"
Oct 30 19:38:28.738071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.738033    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:38:28.788037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.787999    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-kube-api-access-pdmbw\" (UniqueName: \"kubernetes.io/projected/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-kube-api-access-pdmbw\") pod \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\" (UID: \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\") "
Oct 30 19:38:28.788266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.788203    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-plugins-config-volume\") pod \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\" (UID: \"f78e5730-0b9f-45b1-8e5a-16f1d96e3228\") "
Oct 30 19:38:28.788673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.788643    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-plugins-config-volume" pod "f78e5730-0b9f-45b1-8e5a-16f1d96e3228" (UID: "f78e5730-0b9f-45b1-8e5a-16f1d96e3228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:28.789223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.789196    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:28.790538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.790504    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-kube-api-access-pdmbw" pod "f78e5730-0b9f-45b1-8e5a-16f1d96e3228" (UID: "f78e5730-0b9f-45b1-8e5a-16f1d96e3228"). InnerVolumeSpecName "kube-api-access-pdmbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:28.891198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:28.891156    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdmbw\" (UniqueName: \"kubernetes.io/projected/f78e5730-0b9f-45b1-8e5a-16f1d96e3228-kube-api-access-pdmbw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:29.029284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.029258    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:29.036721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.036689    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:29.267865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.267821    6777 generic.go:356] "Generic (PLEG): container finished" podID="f78e5730-0b9f-45b1-8e5a-16f1d96e3228" containerID="066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497" exitCode=0
Oct 30 19:38:29.268364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.267896    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"
Oct 30 19:38:29.268364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.267906    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd" event={"ID":"f78e5730-0b9f-45b1-8e5a-16f1d96e3228","Type":"ContainerDied","Data":"066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497"}
Oct 30 19:38:29.268364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.267973    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd" event={"ID":"f78e5730-0b9f-45b1-8e5a-16f1d96e3228","Type":"ContainerDied","Data":"a4bc135e66a0f4badb15658a138e00c0ea9f8023450b36c6d51c1c61d26545f8"}
Oct 30 19:38:29.268364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.268002    6777 scope.go:122] "RemoveContainer" containerID="066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497"
Oct 30 19:38:29.268608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.268430    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:29.269861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.269841    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:38:29.282943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.282920    6777 scope.go:122] "RemoveContainer" containerID="066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497"
Oct 30 19:38:29.283301 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:38:29.283281    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497\": container with ID starting with 066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497 not found: ID does not exist" containerID="066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497"
Oct 30 19:38:29.283355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.283311    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497"} err="failed to get container status \"066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497\": rpc error: code = NotFound desc = could not find container \"066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497\": container with ID starting with 066ce2a8ee3bcd8f04436e804fccbad90b74dd27c79d7f0ffcc50cd0b07ae497 not found: ID does not exist"
Oct 30 19:38:29.327366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.327328    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"]
Oct 30 19:38:29.336653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.333372    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-76117c-5f9c6f6859-6f7cd"]
Oct 30 19:38:29.497392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.497360    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"]
Oct 30 19:38:29.498362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.498342    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f78e5730-0b9f-45b1-8e5a-16f1d96e3228" containerName="payload-pre-processing"
Oct 30 19:38:29.498455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.498363    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78e5730-0b9f-45b1-8e5a-16f1d96e3228" containerName="payload-pre-processing"
Oct 30 19:38:29.498499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.498493    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78e5730-0b9f-45b1-8e5a-16f1d96e3228" containerName="payload-pre-processing"
Oct 30 19:38:29.498587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.498503    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15081ca-227c-428f-ad9e-a5bac902df6e" containerName="istio-proxy"
Oct 30 19:38:29.501993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.501969    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.506597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.506555    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:38:29.506739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.506626    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:38:29.511861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.511834    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"]
Oct 30 19:38:29.604717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.604626    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mwz\" (UniqueName: \"kubernetes.io/projected/6aad22ba-304a-4f81-bec1-66fd872a8174-kube-api-access-27mwz\") pod \"maas-api-revoke-sub-75a2beca-sql6f\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.604890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.604719    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a2beca-sql6f\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.604890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.604811    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a2beca-sql6f\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.706693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.706661    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mwz\" (UniqueName: \"kubernetes.io/projected/6aad22ba-304a-4f81-bec1-66fd872a8174-kube-api-access-27mwz\") pod \"maas-api-revoke-sub-75a2beca-sql6f\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.706932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.706717    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a2beca-sql6f\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.706932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.706791    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a2beca-sql6f\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.707554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.707532    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a2beca-sql6f\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.709313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.709284    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a2beca-sql6f\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.720463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.720436    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mwz\" (UniqueName: \"kubernetes.io/projected/6aad22ba-304a-4f81-bec1-66fd872a8174-kube-api-access-27mwz\") pod \"maas-api-revoke-sub-75a2beca-sql6f\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:29.820987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:29.820950    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:30.117140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:30.117115    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78e5730-0b9f-45b1-8e5a-16f1d96e3228" path="/var/lib/kubelet/pods/f78e5730-0b9f-45b1-8e5a-16f1d96e3228/volumes"
Oct 30 19:38:30.120504 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:38:30.120467    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aad22ba_304a_4f81_bec1_66fd872a8174.slice/crio-ef932c43759b9f79d8f7e75af567cd41f702b9ff650b1042a0a45707e2db50ba WatchSource:0}: Error finding container ef932c43759b9f79d8f7e75af567cd41f702b9ff650b1042a0a45707e2db50ba: Status 404 returned error can't find the container with id ef932c43759b9f79d8f7e75af567cd41f702b9ff650b1042a0a45707e2db50ba
Oct 30 19:38:30.122048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:30.122020    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"]
Oct 30 19:38:30.280041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:30.280005    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f" event={"ID":"6aad22ba-304a-4f81-bec1-66fd872a8174","Type":"ContainerStarted","Data":"ef932c43759b9f79d8f7e75af567cd41f702b9ff650b1042a0a45707e2db50ba"}
Oct 30 19:38:31.302102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:31.302035    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f" event={"ID":"6aad22ba-304a-4f81-bec1-66fd872a8174","Type":"ContainerStarted","Data":"42507deb2eca97efa0250bc4477959e861dc7f38f235add9981e4f99f2041cf9"}
Oct 30 19:38:31.330362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:31.330309    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f" podStartSLOduration=2.33029411 podStartE2EDuration="2.33029411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:31.326792196 +0000 UTC m=+1893.323707580" watchObservedRunningTime="2026-10-30 19:38:31.33029411 +0000 UTC m=+1893.327209494"
Oct 30 19:38:32.254216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.254183    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"]
Oct 30 19:38:32.258001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.257981    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:32.275397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.275358    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-178bcc-dockercfg-6rr7f\""
Oct 30 19:38:32.275397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.275384    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-178bcc\""
Oct 30 19:38:32.277304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.277276    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"]
Oct 30 19:38:32.318656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.318613    6777 generic.go:356] "Generic (PLEG): container finished" podID="6aad22ba-304a-4f81-bec1-66fd872a8174" containerID="42507deb2eca97efa0250bc4477959e861dc7f38f235add9981e4f99f2041cf9" exitCode=0
Oct 30 19:38:32.319669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.318695    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f" event={"ID":"6aad22ba-304a-4f81-bec1-66fd872a8174","Type":"ContainerDied","Data":"42507deb2eca97efa0250bc4477959e861dc7f38f235add9981e4f99f2041cf9"}
Oct 30 19:38:32.527355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.527245    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc2s9\" (UniqueName: \"kubernetes.io/projected/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-kube-api-access-fc2s9\") pod \"payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr\" (UID: \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:32.542835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.542789    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr\" (UID: \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:32.700163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.663801    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr\" (UID: \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:32.700838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.700790    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc2s9\" (UniqueName: \"kubernetes.io/projected/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-kube-api-access-fc2s9\") pod \"payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr\" (UID: \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:32.712860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.667790    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr\" (UID: \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:32.733160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.733124    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc2s9\" (UniqueName: \"kubernetes.io/projected/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-kube-api-access-fc2s9\") pod \"payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr\" (UID: \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:32.881540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:32.881499    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:33.037603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.037557    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"]
Oct 30 19:38:33.042818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.042786    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-597cded3-pcwjt"]
Oct 30 19:38:33.176560 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:38:33.176522    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2818989_ff6b_4cba_9269_a91dc8d6a9c4.slice/crio-40eb6c4854eecd0d77e4ee1f5459ec9b382fdaaf3108d00b00e3a36cbbd679b5 WatchSource:0}: Error finding container 40eb6c4854eecd0d77e4ee1f5459ec9b382fdaaf3108d00b00e3a36cbbd679b5: Status 404 returned error can't find the container with id 40eb6c4854eecd0d77e4ee1f5459ec9b382fdaaf3108d00b00e3a36cbbd679b5
Oct 30 19:38:33.177329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.177300    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"]
Oct 30 19:38:33.331361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.331323    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr" event={"ID":"d2818989-ff6b-4cba-9269-a91dc8d6a9c4","Type":"ContainerStarted","Data":"40eb6c4854eecd0d77e4ee1f5459ec9b382fdaaf3108d00b00e3a36cbbd679b5"}
Oct 30 19:38:33.508048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.508022    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:33.536118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.536082    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-xks-ca\") pod \"6aad22ba-304a-4f81-bec1-66fd872a8174\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") "
Oct 30 19:38:33.540107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.537306    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aad22ba-304a-4f81-bec1-66fd872a8174-kube-api-access-27mwz\" (UniqueName: \"kubernetes.io/projected/6aad22ba-304a-4f81-bec1-66fd872a8174-kube-api-access-27mwz\") pod \"6aad22ba-304a-4f81-bec1-66fd872a8174\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") "
Oct 30 19:38:33.540107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.537506    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-openshift-ca\") pod \"6aad22ba-304a-4f81-bec1-66fd872a8174\" (UID: \"6aad22ba-304a-4f81-bec1-66fd872a8174\") "
Oct 30 19:38:33.540107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.539730    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-xks-ca" pod "6aad22ba-304a-4f81-bec1-66fd872a8174" (UID: "6aad22ba-304a-4f81-bec1-66fd872a8174"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:33.540381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.540299    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aad22ba-304a-4f81-bec1-66fd872a8174-kube-api-access-27mwz" pod "6aad22ba-304a-4f81-bec1-66fd872a8174" (UID: "6aad22ba-304a-4f81-bec1-66fd872a8174"). InnerVolumeSpecName "kube-api-access-27mwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:33.542288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.541232    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-openshift-ca" pod "6aad22ba-304a-4f81-bec1-66fd872a8174" (UID: "6aad22ba-304a-4f81-bec1-66fd872a8174"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:33.641381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.641340    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:33.641381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.641375    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aad22ba-304a-4f81-bec1-66fd872a8174-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:33.641381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:33.641388    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27mwz\" (UniqueName: \"kubernetes.io/projected/6aad22ba-304a-4f81-bec1-66fd872a8174-kube-api-access-27mwz\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:34.115561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:34.115525    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d096e9-4314-49e3-a6bb-faa6a49e5563" path="/var/lib/kubelet/pods/c6d096e9-4314-49e3-a6bb-faa6a49e5563/volumes"
Oct 30 19:38:34.342268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:34.342238    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"
Oct 30 19:38:34.342725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:34.342267    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f" event={"ID":"6aad22ba-304a-4f81-bec1-66fd872a8174","Type":"ContainerDied","Data":"ef932c43759b9f79d8f7e75af567cd41f702b9ff650b1042a0a45707e2db50ba"}
Oct 30 19:38:34.342725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:34.342330    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef932c43759b9f79d8f7e75af567cd41f702b9ff650b1042a0a45707e2db50ba"
Oct 30 19:38:34.345174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:34.345137    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr" event={"ID":"d2818989-ff6b-4cba-9269-a91dc8d6a9c4","Type":"ContainerStarted","Data":"3aea1afc467c478b748888708dedacf472263ff7720afaa5eee7014813eef6be"}
Oct 30 19:38:34.345346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:34.345215    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:34.367703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:34.367592    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr" podStartSLOduration=2.367578141 podStartE2EDuration="2.367578141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:34.365190305 +0000 UTC m=+1896.362105708" watchObservedRunningTime="2026-10-30 19:38:34.367578141 +0000 UTC m=+1896.364493563"
Oct 30 19:38:39.838779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.838696    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"]
Oct 30 19:38:39.839649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.839633    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6aad22ba-304a-4f81-bec1-66fd872a8174" containerName="revoke-keys"
Oct 30 19:38:39.839697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.839650    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aad22ba-304a-4f81-bec1-66fd872a8174" containerName="revoke-keys"
Oct 30 19:38:39.839779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.839770    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aad22ba-304a-4f81-bec1-66fd872a8174" containerName="revoke-keys"
Oct 30 19:38:39.842943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.842927    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:39.847611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.847586    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:38:39.847794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.847772    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:38:39.868857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.868827    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"]
Oct 30 19:38:39.904305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.904270    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1558337b-rn75n\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:39.904522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.904310    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1558337b-rn75n\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:39.904649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:39.904624    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zss8j\" (UniqueName: \"kubernetes.io/projected/43a8dda2-0d82-4069-b850-960a9f30d094-kube-api-access-zss8j\") pod \"maas-api-revoke-sub-1558337b-rn75n\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:40.006200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:40.006110    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1558337b-rn75n\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:40.006200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:40.006157    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1558337b-rn75n\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:40.006477 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:40.006385    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zss8j\" (UniqueName: \"kubernetes.io/projected/43a8dda2-0d82-4069-b850-960a9f30d094-kube-api-access-zss8j\") pod \"maas-api-revoke-sub-1558337b-rn75n\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:40.007846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:40.007822    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1558337b-rn75n\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:40.008761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:40.008742    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1558337b-rn75n\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:40.019398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:40.019374    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zss8j\" (UniqueName: \"kubernetes.io/projected/43a8dda2-0d82-4069-b850-960a9f30d094-kube-api-access-zss8j\") pod \"maas-api-revoke-sub-1558337b-rn75n\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:40.157253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:40.157158    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:40.453626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:40.453182    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"]
Oct 30 19:38:41.418648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:41.418615    6777 generic.go:356] "Generic (PLEG): container finished" podID="43a8dda2-0d82-4069-b850-960a9f30d094" containerID="71b3962a91cef926b9b8891dd84c0d690e75fe438e86e4bad1e5a15569f00c6e" exitCode=0
Oct 30 19:38:41.419071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:41.418694    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n" event={"ID":"43a8dda2-0d82-4069-b850-960a9f30d094","Type":"ContainerDied","Data":"71b3962a91cef926b9b8891dd84c0d690e75fe438e86e4bad1e5a15569f00c6e"}
Oct 30 19:38:41.419071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:41.418776    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n" event={"ID":"43a8dda2-0d82-4069-b850-960a9f30d094","Type":"ContainerStarted","Data":"7ca5ef77f41696481c6aba308208447d3080f3e0ac6062e91add2339d925e1e2"}
Oct 30 19:38:42.672192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.672165    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:42.771697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.771604    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-xks-ca\") pod \"43a8dda2-0d82-4069-b850-960a9f30d094\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") "
Oct 30 19:38:42.771697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.771695    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a8dda2-0d82-4069-b850-960a9f30d094-kube-api-access-zss8j\" (UniqueName: \"kubernetes.io/projected/43a8dda2-0d82-4069-b850-960a9f30d094-kube-api-access-zss8j\") pod \"43a8dda2-0d82-4069-b850-960a9f30d094\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") "
Oct 30 19:38:42.771905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.771884    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-openshift-ca\") pod \"43a8dda2-0d82-4069-b850-960a9f30d094\" (UID: \"43a8dda2-0d82-4069-b850-960a9f30d094\") "
Oct 30 19:38:42.772331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.772299    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-openshift-ca" pod "43a8dda2-0d82-4069-b850-960a9f30d094" (UID: "43a8dda2-0d82-4069-b850-960a9f30d094"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:42.772814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.772794    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.774374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.774342    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a8dda2-0d82-4069-b850-960a9f30d094-kube-api-access-zss8j" pod "43a8dda2-0d82-4069-b850-960a9f30d094" (UID: "43a8dda2-0d82-4069-b850-960a9f30d094"). InnerVolumeSpecName "kube-api-access-zss8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:42.774374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.774364    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-xks-ca" pod "43a8dda2-0d82-4069-b850-960a9f30d094" (UID: "43a8dda2-0d82-4069-b850-960a9f30d094"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:42.874761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.874726    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8dda2-0d82-4069-b850-960a9f30d094-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.874761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:42.874755    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zss8j\" (UniqueName: \"kubernetes.io/projected/43a8dda2-0d82-4069-b850-960a9f30d094-kube-api-access-zss8j\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:43.437203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:43.437173    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"
Oct 30 19:38:43.437390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:43.437177    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n" event={"ID":"43a8dda2-0d82-4069-b850-960a9f30d094","Type":"ContainerDied","Data":"7ca5ef77f41696481c6aba308208447d3080f3e0ac6062e91add2339d925e1e2"}
Oct 30 19:38:43.437390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:43.437304    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca5ef77f41696481c6aba308208447d3080f3e0ac6062e91add2339d925e1e2"
Oct 30 19:38:43.645536 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:38:43.645498    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:38:45.356835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:45.356804    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:38:48.397256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.397224    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"]
Oct 30 19:38:48.398148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.398130    6777 cpu_manager.go:414] "containerMap: removing container" podUID="43a8dda2-0d82-4069-b850-960a9f30d094" containerName="revoke-keys"
Oct 30 19:38:48.398148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.398148    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a8dda2-0d82-4069-b850-960a9f30d094" containerName="revoke-keys"
Oct 30 19:38:48.398296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.398286    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a8dda2-0d82-4069-b850-960a9f30d094" containerName="revoke-keys"
Oct 30 19:38:48.401308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.401290    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.405665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.405643    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:38:48.406405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.406035    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:38:48.410635 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.410610    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"]
Oct 30 19:38:48.473883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.473841    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfc72\" (UniqueName: \"kubernetes.io/projected/3450d757-8cf5-425e-adb3-4af99cca13dc-kube-api-access-tfc72\") pod \"maas-api-revoke-sub-35ba8590-6tq5x\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.474347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.474323    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ba8590-6tq5x\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.474500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.474358    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ba8590-6tq5x\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.576192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.576134    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfc72\" (UniqueName: \"kubernetes.io/projected/3450d757-8cf5-425e-adb3-4af99cca13dc-kube-api-access-tfc72\") pod \"maas-api-revoke-sub-35ba8590-6tq5x\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.576414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.576357    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ba8590-6tq5x\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.576414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.576384    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ba8590-6tq5x\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.577116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.577041    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ba8590-6tq5x\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.579375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.579350    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ba8590-6tq5x\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.589444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.589418    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfc72\" (UniqueName: \"kubernetes.io/projected/3450d757-8cf5-425e-adb3-4af99cca13dc-kube-api-access-tfc72\") pod \"maas-api-revoke-sub-35ba8590-6tq5x\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:48.718817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:48.718734    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:49.008833 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:38:49.008803    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3450d757_8cf5_425e_adb3_4af99cca13dc.slice/crio-c32a1680df5b606b408bb84f3b7848ae5ea9a76270ba52d90c78897450dc73d8 WatchSource:0}: Error finding container c32a1680df5b606b408bb84f3b7848ae5ea9a76270ba52d90c78897450dc73d8: Status 404 returned error can't find the container with id c32a1680df5b606b408bb84f3b7848ae5ea9a76270ba52d90c78897450dc73d8
Oct 30 19:38:49.009254 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:49.009230    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"]
Oct 30 19:38:49.515045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:49.515005    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x" event={"ID":"3450d757-8cf5-425e-adb3-4af99cca13dc","Type":"ContainerStarted","Data":"4b1dc63c99f95b31992598839f4ab3be140e439dc9ed000a1a253c6f53383074"}
Oct 30 19:38:49.515542 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:49.515078    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x" event={"ID":"3450d757-8cf5-425e-adb3-4af99cca13dc","Type":"ContainerStarted","Data":"c32a1680df5b606b408bb84f3b7848ae5ea9a76270ba52d90c78897450dc73d8"}
Oct 30 19:38:49.556456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:49.556393    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x" podStartSLOduration=1.556376955 podStartE2EDuration="1.556376955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:49.554662142 +0000 UTC m=+1911.551577525" watchObservedRunningTime="2026-10-30 19:38:49.556376955 +0000 UTC m=+1911.553292338"
Oct 30 19:38:50.525332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:50.525215    6777 generic.go:356] "Generic (PLEG): container finished" podID="3450d757-8cf5-425e-adb3-4af99cca13dc" containerID="4b1dc63c99f95b31992598839f4ab3be140e439dc9ed000a1a253c6f53383074" exitCode=0
Oct 30 19:38:50.525332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:50.525297    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x" event={"ID":"3450d757-8cf5-425e-adb3-4af99cca13dc","Type":"ContainerDied","Data":"4b1dc63c99f95b31992598839f4ab3be140e439dc9ed000a1a253c6f53383074"}
Oct 30 19:38:51.822006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:51.821971    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:51.861346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:51.861241    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"]
Oct 30 19:38:51.963080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:51.956705    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-openshift-ca\") pod \"3450d757-8cf5-425e-adb3-4af99cca13dc\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") "
Oct 30 19:38:51.963080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:51.956879    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3450d757-8cf5-425e-adb3-4af99cca13dc-kube-api-access-tfc72\" (UniqueName: \"kubernetes.io/projected/3450d757-8cf5-425e-adb3-4af99cca13dc-kube-api-access-tfc72\") pod \"3450d757-8cf5-425e-adb3-4af99cca13dc\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") "
Oct 30 19:38:51.963080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:51.957176    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-xks-ca\") pod \"3450d757-8cf5-425e-adb3-4af99cca13dc\" (UID: \"3450d757-8cf5-425e-adb3-4af99cca13dc\") "
Oct 30 19:38:52.135149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:52.108828    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-openshift-ca" pod "3450d757-8cf5-425e-adb3-4af99cca13dc" (UID: "3450d757-8cf5-425e-adb3-4af99cca13dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:52.150568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:52.149660    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:52.181845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:52.180788    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3450d757-8cf5-425e-adb3-4af99cca13dc-kube-api-access-tfc72" pod "3450d757-8cf5-425e-adb3-4af99cca13dc" (UID: "3450d757-8cf5-425e-adb3-4af99cca13dc"). InnerVolumeSpecName "kube-api-access-tfc72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:52.197657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:52.184970    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-xks-ca" pod "3450d757-8cf5-425e-adb3-4af99cca13dc" (UID: "3450d757-8cf5-425e-adb3-4af99cca13dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:52.264559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:52.264029    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfc72\" (UniqueName: \"kubernetes.io/projected/3450d757-8cf5-425e-adb3-4af99cca13dc-kube-api-access-tfc72\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:52.264559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:52.264082    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3450d757-8cf5-425e-adb3-4af99cca13dc-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:52.289845 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:38:52.289802    6777 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3450d757_8cf5_425e_adb3_4af99cca13dc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:38:52.597632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:52.597596    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x" event={"ID":"3450d757-8cf5-425e-adb3-4af99cca13dc","Type":"ContainerDied","Data":"c32a1680df5b606b408bb84f3b7848ae5ea9a76270ba52d90c78897450dc73d8"}
Oct 30 19:38:52.597632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:52.597631    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"
Oct 30 19:38:52.597856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:52.597654    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c32a1680df5b606b408bb84f3b7848ae5ea9a76270ba52d90c78897450dc73d8"
Oct 30 19:38:54.888657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:54.888620    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:38:59.097308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.097268    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"]
Oct 30 19:38:59.098533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.098504    6777 cpu_manager.go:414] "containerMap: removing container" podUID="3450d757-8cf5-425e-adb3-4af99cca13dc" containerName="revoke-keys"
Oct 30 19:38:59.098533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.098528    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3450d757-8cf5-425e-adb3-4af99cca13dc" containerName="revoke-keys"
Oct 30 19:38:59.099244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.098778    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="3450d757-8cf5-425e-adb3-4af99cca13dc" containerName="revoke-keys"
Oct 30 19:38:59.103508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.103484    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.107543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.107518    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:38:59.108775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.108752    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:38:59.110051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.110026    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"]
Oct 30 19:38:59.153758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.153699    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8mlf\" (UniqueName: \"kubernetes.io/projected/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-kube-api-access-z8mlf\") pod \"maas-api-revoke-sub-ebd7e582-nfc2d\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.153977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.153844    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd7e582-nfc2d\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.153977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.153910    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd7e582-nfc2d\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.256496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.256462    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8mlf\" (UniqueName: \"kubernetes.io/projected/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-kube-api-access-z8mlf\") pod \"maas-api-revoke-sub-ebd7e582-nfc2d\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.256678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.256535    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd7e582-nfc2d\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.256678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.256573    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd7e582-nfc2d\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.257338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.257311    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd7e582-nfc2d\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.260699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.260671    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd7e582-nfc2d\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.269173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.269152    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8mlf\" (UniqueName: \"kubernetes.io/projected/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-kube-api-access-z8mlf\") pod \"maas-api-revoke-sub-ebd7e582-nfc2d\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.425950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.425859    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:38:59.872701 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:38:59.872663    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e85a7b4_91c1_446b_9160_1ca4d7212bfa.slice/crio-67a7f511050988e7b13c194b28a352d43192b252ebb06e68a0ab4cc989f2111d WatchSource:0}: Error finding container 67a7f511050988e7b13c194b28a352d43192b252ebb06e68a0ab4cc989f2111d: Status 404 returned error can't find the container with id 67a7f511050988e7b13c194b28a352d43192b252ebb06e68a0ab4cc989f2111d
Oct 30 19:38:59.873005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:38:59.872986    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"]
Oct 30 19:39:00.743850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:00.743812    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d" event={"ID":"7e85a7b4-91c1-446b-9160-1ca4d7212bfa","Type":"ContainerStarted","Data":"8392bb7d25def63db052cd228f39dfb832c408f99f694ba30b4fda055c752948"}
Oct 30 19:39:00.744317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:00.743880    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d" event={"ID":"7e85a7b4-91c1-446b-9160-1ca4d7212bfa","Type":"ContainerStarted","Data":"67a7f511050988e7b13c194b28a352d43192b252ebb06e68a0ab4cc989f2111d"}
Oct 30 19:39:00.768353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:00.768297    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d" podStartSLOduration=1.768281055 podStartE2EDuration="1.768281055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:00.764294509 +0000 UTC m=+1922.761209904" watchObservedRunningTime="2026-10-30 19:39:00.768281055 +0000 UTC m=+1922.765196437"
Oct 30 19:39:01.756197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:01.756161    6777 generic.go:356] "Generic (PLEG): container finished" podID="7e85a7b4-91c1-446b-9160-1ca4d7212bfa" containerID="8392bb7d25def63db052cd228f39dfb832c408f99f694ba30b4fda055c752948" exitCode=0
Oct 30 19:39:01.756626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:01.756256    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d" event={"ID":"7e85a7b4-91c1-446b-9160-1ca4d7212bfa","Type":"ContainerDied","Data":"8392bb7d25def63db052cd228f39dfb832c408f99f694ba30b4fda055c752948"}
Oct 30 19:39:03.022605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.022575    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:39:03.101301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.101200    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-kube-api-access-z8mlf\" (UniqueName: \"kubernetes.io/projected/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-kube-api-access-z8mlf\") pod \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") "
Oct 30 19:39:03.102269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.101477    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-openshift-ca\") pod \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") "
Oct 30 19:39:03.102269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.101579    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-xks-ca\") pod \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\" (UID: \"7e85a7b4-91c1-446b-9160-1ca4d7212bfa\") "
Oct 30 19:39:03.102269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.101989    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-openshift-ca" pod "7e85a7b4-91c1-446b-9160-1ca4d7212bfa" (UID: "7e85a7b4-91c1-446b-9160-1ca4d7212bfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:03.103830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.102505    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:03.105098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.105043    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-xks-ca" pod "7e85a7b4-91c1-446b-9160-1ca4d7212bfa" (UID: "7e85a7b4-91c1-446b-9160-1ca4d7212bfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:03.105262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.105236    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-kube-api-access-z8mlf" pod "7e85a7b4-91c1-446b-9160-1ca4d7212bfa" (UID: "7e85a7b4-91c1-446b-9160-1ca4d7212bfa"). InnerVolumeSpecName "kube-api-access-z8mlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:03.204655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.204615    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:03.204655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.204650    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8mlf\" (UniqueName: \"kubernetes.io/projected/7e85a7b4-91c1-446b-9160-1ca4d7212bfa-kube-api-access-z8mlf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:03.817043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.817008    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"
Oct 30 19:39:03.817043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.817016    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d" event={"ID":"7e85a7b4-91c1-446b-9160-1ca4d7212bfa","Type":"ContainerDied","Data":"67a7f511050988e7b13c194b28a352d43192b252ebb06e68a0ab4cc989f2111d"}
Oct 30 19:39:03.817043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.817089    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a7f511050988e7b13c194b28a352d43192b252ebb06e68a0ab4cc989f2111d"
Oct 30 19:39:03.883052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:03.882988    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"]
Oct 30 19:39:04.620129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.620049    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"]
Oct 30 19:39:04.622832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.622794    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7e85a7b4-91c1-446b-9160-1ca4d7212bfa" containerName="revoke-keys"
Oct 30 19:39:04.623113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.623093    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e85a7b4-91c1-446b-9160-1ca4d7212bfa" containerName="revoke-keys"
Oct 30 19:39:04.623754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.623730    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e85a7b4-91c1-446b-9160-1ca4d7212bfa" containerName="revoke-keys"
Oct 30 19:39:04.638243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.638211    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.639294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.639227    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"]
Oct 30 19:39:04.659400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.659351    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:04.659608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.659445    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:39:04.660894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.660860    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c4aba99-nrdbb\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.661042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.660940    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p254v\" (UniqueName: \"kubernetes.io/projected/a71ab85f-50a8-4ef0-9490-08579fca34bf-kube-api-access-p254v\") pod \"maas-api-revoke-sub-0c4aba99-nrdbb\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.661313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.661277    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c4aba99-nrdbb\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.777217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.773575    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p254v\" (UniqueName: \"kubernetes.io/projected/a71ab85f-50a8-4ef0-9490-08579fca34bf-kube-api-access-p254v\") pod \"maas-api-revoke-sub-0c4aba99-nrdbb\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.777217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.773856    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c4aba99-nrdbb\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.777217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.774219    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c4aba99-nrdbb\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.797081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.784651    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c4aba99-nrdbb\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.797081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.795902    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c4aba99-nrdbb\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.806605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.806571    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p254v\" (UniqueName: \"kubernetes.io/projected/a71ab85f-50a8-4ef0-9490-08579fca34bf-kube-api-access-p254v\") pod \"maas-api-revoke-sub-0c4aba99-nrdbb\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:04.969932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:04.969827    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:05.274411 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:39:05.274371    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71ab85f_50a8_4ef0_9490_08579fca34bf.slice/crio-7d7fc1e3073ce0feea2b4bf2611f89a05a10a9d4d88985f485629e1568284db6 WatchSource:0}: Error finding container 7d7fc1e3073ce0feea2b4bf2611f89a05a10a9d4d88985f485629e1568284db6: Status 404 returned error can't find the container with id 7d7fc1e3073ce0feea2b4bf2611f89a05a10a9d4d88985f485629e1568284db6
Oct 30 19:39:05.275147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:05.275112    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"]
Oct 30 19:39:05.841454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:05.841348    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb" event={"ID":"a71ab85f-50a8-4ef0-9490-08579fca34bf","Type":"ContainerStarted","Data":"7c46a70d832f8563b660f939024f0919a17500d53ad559cc45d960402777c297"}
Oct 30 19:39:05.841454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:05.841409    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb" event={"ID":"a71ab85f-50a8-4ef0-9490-08579fca34bf","Type":"ContainerStarted","Data":"7d7fc1e3073ce0feea2b4bf2611f89a05a10a9d4d88985f485629e1568284db6"}
Oct 30 19:39:05.865514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:05.865452    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb" podStartSLOduration=1.8654354739999999 podStartE2EDuration="1.865435474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:05.862025746 +0000 UTC m=+1927.858941129" watchObservedRunningTime="2026-10-30 19:39:05.865435474 +0000 UTC m=+1927.862350857"
Oct 30 19:39:06.852620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:06.852578    6777 generic.go:356] "Generic (PLEG): container finished" podID="a71ab85f-50a8-4ef0-9490-08579fca34bf" containerID="7c46a70d832f8563b660f939024f0919a17500d53ad559cc45d960402777c297" exitCode=0
Oct 30 19:39:06.853143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:06.852649    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb" event={"ID":"a71ab85f-50a8-4ef0-9490-08579fca34bf","Type":"ContainerDied","Data":"7c46a70d832f8563b660f939024f0919a17500d53ad559cc45d960402777c297"}
Oct 30 19:39:06.904847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:06.904794    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main" containerID="cri-o://4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065" gracePeriod=45
Oct 30 19:39:07.290726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.290700    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:39:07.294215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.294192    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-dshm\") pod \"0faac039-491d-4dd9-aac1-5309b17b599e\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") "
Oct 30 19:39:07.294326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.294240    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0faac039-491d-4dd9-aac1-5309b17b599e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0faac039-491d-4dd9-aac1-5309b17b599e-tls-certs\") pod \"0faac039-491d-4dd9-aac1-5309b17b599e\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") "
Oct 30 19:39:07.294326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.294282    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0faac039-491d-4dd9-aac1-5309b17b599e-kube-api-access-mqv5b\" (UniqueName: \"kubernetes.io/projected/0faac039-491d-4dd9-aac1-5309b17b599e-kube-api-access-mqv5b\") pod \"0faac039-491d-4dd9-aac1-5309b17b599e\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") "
Oct 30 19:39:07.294445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.294430    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-home\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-home\") pod \"0faac039-491d-4dd9-aac1-5309b17b599e\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") "
Oct 30 19:39:07.294519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.294504    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-tmp-dir\") pod \"0faac039-491d-4dd9-aac1-5309b17b599e\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") "
Oct 30 19:39:07.294585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.294548    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-model-cache\") pod \"0faac039-491d-4dd9-aac1-5309b17b599e\" (UID: \"0faac039-491d-4dd9-aac1-5309b17b599e\") "
Oct 30 19:39:07.294744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.294714    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-home" pod "0faac039-491d-4dd9-aac1-5309b17b599e" (UID: "0faac039-491d-4dd9-aac1-5309b17b599e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:07.294744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.294723    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-tmp-dir" pod "0faac039-491d-4dd9-aac1-5309b17b599e" (UID: "0faac039-491d-4dd9-aac1-5309b17b599e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:07.295012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.294976    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-model-cache" pod "0faac039-491d-4dd9-aac1-5309b17b599e" (UID: "0faac039-491d-4dd9-aac1-5309b17b599e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:07.295439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.295417    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:07.295511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.295442    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:07.295511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.295456    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:07.297766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.297734    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0faac039-491d-4dd9-aac1-5309b17b599e-kube-api-access-mqv5b" pod "0faac039-491d-4dd9-aac1-5309b17b599e" (UID: "0faac039-491d-4dd9-aac1-5309b17b599e"). InnerVolumeSpecName "kube-api-access-mqv5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:07.297861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.297832    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0faac039-491d-4dd9-aac1-5309b17b599e-tls-certs" pod "0faac039-491d-4dd9-aac1-5309b17b599e" (UID: "0faac039-491d-4dd9-aac1-5309b17b599e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:07.297908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.297855    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-dshm" pod "0faac039-491d-4dd9-aac1-5309b17b599e" (UID: "0faac039-491d-4dd9-aac1-5309b17b599e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:07.397561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.397522    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0faac039-491d-4dd9-aac1-5309b17b599e-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:07.397561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.397555    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqv5b\" (UniqueName: \"kubernetes.io/projected/0faac039-491d-4dd9-aac1-5309b17b599e-kube-api-access-mqv5b\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:07.397561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.397567    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faac039-491d-4dd9-aac1-5309b17b599e-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:07.906180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.903961    6777 generic.go:356] "Generic (PLEG): container finished" podID="0faac039-491d-4dd9-aac1-5309b17b599e" containerID="4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065" exitCode=0
Oct 30 19:39:07.906180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.904927    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"
Oct 30 19:39:07.909277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.909237    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" event={"ID":"0faac039-491d-4dd9-aac1-5309b17b599e","Type":"ContainerDied","Data":"4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065"}
Oct 30 19:39:07.910031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.909995    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9" event={"ID":"0faac039-491d-4dd9-aac1-5309b17b599e","Type":"ContainerDied","Data":"7c2002022a089e603280a7f3673665bf7bca15e194f29dcc2f1af1bba68c7198"}
Oct 30 19:39:07.911760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.910053    6777 scope.go:122] "RemoveContainer" containerID="4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065"
Oct 30 19:39:07.932051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.932026    6777 scope.go:122] "RemoveContainer" containerID="4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065"
Oct 30 19:39:07.933370 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:39:07.933329    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065\": container with ID starting with 4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065 not found: ID does not exist" containerID="4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065"
Oct 30 19:39:07.933534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.933394    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065"} err="failed to get container status \"4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065\": rpc error: code = NotFound desc = could not find container \"4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065\": container with ID starting with 4d823fbcc483345f26e11ccc6b1c04077e6767824f3360f7b011f887d0dec065 not found: ID does not exist"
Oct 30 19:39:07.952265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.952209    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"]
Oct 30 19:39:07.967585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:07.966513    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/auth-test-model-69c573-kserve-868574f8f7-fl4b9"]
Oct 30 19:39:08.120666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.120621    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" path="/var/lib/kubelet/pods/0faac039-491d-4dd9-aac1-5309b17b599e/volumes"
Oct 30 19:39:08.137088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.137045    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:08.214708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.214601    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71ab85f-50a8-4ef0-9490-08579fca34bf-kube-api-access-p254v\" (UniqueName: \"kubernetes.io/projected/a71ab85f-50a8-4ef0-9490-08579fca34bf-kube-api-access-p254v\") pod \"a71ab85f-50a8-4ef0-9490-08579fca34bf\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") "
Oct 30 19:39:08.214898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.214766    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-xks-ca\") pod \"a71ab85f-50a8-4ef0-9490-08579fca34bf\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") "
Oct 30 19:39:08.215166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.215140    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-openshift-ca\") pod \"a71ab85f-50a8-4ef0-9490-08579fca34bf\" (UID: \"a71ab85f-50a8-4ef0-9490-08579fca34bf\") "
Oct 30 19:39:08.215700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.215663    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-openshift-ca" pod "a71ab85f-50a8-4ef0-9490-08579fca34bf" (UID: "a71ab85f-50a8-4ef0-9490-08579fca34bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:08.216298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.216258    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:08.220664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.220629    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-xks-ca" pod "a71ab85f-50a8-4ef0-9490-08579fca34bf" (UID: "a71ab85f-50a8-4ef0-9490-08579fca34bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:08.222629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.222597    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71ab85f-50a8-4ef0-9490-08579fca34bf-kube-api-access-p254v" pod "a71ab85f-50a8-4ef0-9490-08579fca34bf" (UID: "a71ab85f-50a8-4ef0-9490-08579fca34bf"). InnerVolumeSpecName "kube-api-access-p254v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:08.318435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.318371    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p254v\" (UniqueName: \"kubernetes.io/projected/a71ab85f-50a8-4ef0-9490-08579fca34bf-kube-api-access-p254v\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:08.318435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.318427    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71ab85f-50a8-4ef0-9490-08579fca34bf-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:08.918377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.918340    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb" event={"ID":"a71ab85f-50a8-4ef0-9490-08579fca34bf","Type":"ContainerDied","Data":"7d7fc1e3073ce0feea2b4bf2611f89a05a10a9d4d88985f485629e1568284db6"}
Oct 30 19:39:08.918377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.918364    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"
Oct 30 19:39:08.918825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:08.918429    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d7fc1e3073ce0feea2b4bf2611f89a05a10a9d4d88985f485629e1568284db6"
Oct 30 19:39:09.410258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.410219    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"]
Oct 30 19:39:09.411231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.411212    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a71ab85f-50a8-4ef0-9490-08579fca34bf" containerName="revoke-keys"
Oct 30 19:39:09.411289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.411233    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71ab85f-50a8-4ef0-9490-08579fca34bf" containerName="revoke-keys"
Oct 30 19:39:09.411289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.411249    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main"
Oct 30 19:39:09.411289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.411254    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main"
Oct 30 19:39:09.411415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.411404    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71ab85f-50a8-4ef0-9490-08579fca34bf" containerName="revoke-keys"
Oct 30 19:39:09.411450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.411417    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0faac039-491d-4dd9-aac1-5309b17b599e" containerName="main"
Oct 30 19:39:09.415729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.415704    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.420747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.420714    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:09.420935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.420778    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:39:09.424895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.424860    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"]
Oct 30 19:39:09.444495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.444448    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7edc0d-zqfsh\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.444741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.444617    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvr7b\" (UniqueName: \"kubernetes.io/projected/836fdc23-6f7d-4a60-b829-d2969da31873-kube-api-access-zvr7b\") pod \"maas-api-revoke-sub-5e7edc0d-zqfsh\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.444741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.444656    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7edc0d-zqfsh\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.550597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.549171    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvr7b\" (UniqueName: \"kubernetes.io/projected/836fdc23-6f7d-4a60-b829-d2969da31873-kube-api-access-zvr7b\") pod \"maas-api-revoke-sub-5e7edc0d-zqfsh\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.550597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.549237    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7edc0d-zqfsh\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.555084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.552943    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7edc0d-zqfsh\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.563731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.563497    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7edc0d-zqfsh\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.588567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.587157    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7edc0d-zqfsh\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.594232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.594159    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvr7b\" (UniqueName: \"kubernetes.io/projected/836fdc23-6f7d-4a60-b829-d2969da31873-kube-api-access-zvr7b\") pod \"maas-api-revoke-sub-5e7edc0d-zqfsh\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:09.747926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:09.747831    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:10.043814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.043777    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"]
Oct 30 19:39:10.050822 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:39:10.050789    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836fdc23_6f7d_4a60_b829_d2969da31873.slice/crio-ff949cb11cc8251e33c547cf8c75d55e60465a06599e8a00a0cd79f79dcfd0e1 WatchSource:0}: Error finding container ff949cb11cc8251e33c547cf8c75d55e60465a06599e8a00a0cd79f79dcfd0e1: Status 404 returned error can't find the container with id ff949cb11cc8251e33c547cf8c75d55e60465a06599e8a00a0cd79f79dcfd0e1
Oct 30 19:39:10.518041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.518003    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"]
Oct 30 19:39:10.525758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.525719    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.533975 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.533927    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-69c573\"/\"openshift-service-ca.crt\""
Oct 30 19:39:10.534223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.534203    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-69c573\"/\"default-dockercfg-gzflj\""
Oct 30 19:39:10.534306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.534294    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-69c573\"/\"sub-test-model-69c573-kserve-self-signed-certs\""
Oct 30 19:39:10.536757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.535605    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-69c573\"/\"kube-root-ca.crt\""
Oct 30 19:39:10.541588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.541561    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"]
Oct 30 19:39:10.582439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.582400    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-home\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.582673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.582576    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-model-cache\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.582774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.582746    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-tmp-dir\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.583149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.583122    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-dshm\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.583286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.583268    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5sw\" (UniqueName: \"kubernetes.io/projected/d89e2513-3b72-4674-bee1-ed6632d75c2b-kube-api-access-nt5sw\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.583344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.583306    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89e2513-3b72-4674-bee1-ed6632d75c2b-tls-certs\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.685392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.685290    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-dshm\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.685564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.685426    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5sw\" (UniqueName: \"kubernetes.io/projected/d89e2513-3b72-4674-bee1-ed6632d75c2b-kube-api-access-nt5sw\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.685564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.685452    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89e2513-3b72-4674-bee1-ed6632d75c2b-tls-certs\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.685564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.685486    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-home\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.685747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.685599    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-model-cache\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.685747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.685724    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-tmp-dir\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.685924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.685894    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-home\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.686017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.685962    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-model-cache\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.686143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.686111    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-tmp-dir\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.688009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.687981    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-dshm\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.688312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.688293    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89e2513-3b72-4674-bee1-ed6632d75c2b-tls-certs\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.699333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.699297    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5sw\" (UniqueName: \"kubernetes.io/projected/d89e2513-3b72-4674-bee1-ed6632d75c2b-kube-api-access-nt5sw\") pod \"sub-test-model-69c573-kserve-7f49d7d89f-7fhdw\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") " pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.852384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.852040    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:10.979164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.979122    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh" event={"ID":"836fdc23-6f7d-4a60-b829-d2969da31873","Type":"ContainerStarted","Data":"1ce484c0a3f329f49e10ae4c3b32948229d15c89179a9ca0d55e29d8603388aa"}
Oct 30 19:39:10.979445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:10.979421    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh" event={"ID":"836fdc23-6f7d-4a60-b829-d2969da31873","Type":"ContainerStarted","Data":"ff949cb11cc8251e33c547cf8c75d55e60465a06599e8a00a0cd79f79dcfd0e1"}
Oct 30 19:39:11.014264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:11.014211    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh" podStartSLOduration=2.014194464 podStartE2EDuration="2.014194464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:11.000542897 +0000 UTC m=+1932.997458296" watchObservedRunningTime="2026-10-30 19:39:11.014194464 +0000 UTC m=+1933.011109848"
Oct 30 19:39:11.272972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:11.272938    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"]
Oct 30 19:39:11.294268 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:39:11.294232    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd89e2513_3b72_4674_bee1_ed6632d75c2b.slice/crio-2a2b2dc6b9345e883642d5857a18c08dfa1bd2412f1a22099c5d50a26caa697f WatchSource:0}: Error finding container 2a2b2dc6b9345e883642d5857a18c08dfa1bd2412f1a22099c5d50a26caa697f: Status 404 returned error can't find the container with id 2a2b2dc6b9345e883642d5857a18c08dfa1bd2412f1a22099c5d50a26caa697f
Oct 30 19:39:11.991042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:11.990908    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" event={"ID":"d89e2513-3b72-4674-bee1-ed6632d75c2b","Type":"ContainerStarted","Data":"73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49"}
Oct 30 19:39:11.991042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:11.990958    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" event={"ID":"d89e2513-3b72-4674-bee1-ed6632d75c2b","Type":"ContainerStarted","Data":"2a2b2dc6b9345e883642d5857a18c08dfa1bd2412f1a22099c5d50a26caa697f"}
Oct 30 19:39:11.993158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:11.993126    6777 generic.go:356] "Generic (PLEG): container finished" podID="836fdc23-6f7d-4a60-b829-d2969da31873" containerID="1ce484c0a3f329f49e10ae4c3b32948229d15c89179a9ca0d55e29d8603388aa" exitCode=0
Oct 30 19:39:11.993303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:11.993206    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh" event={"ID":"836fdc23-6f7d-4a60-b829-d2969da31873","Type":"ContainerDied","Data":"1ce484c0a3f329f49e10ae4c3b32948229d15c89179a9ca0d55e29d8603388aa"}
Oct 30 19:39:12.016845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:12.016786    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podStartSLOduration=2.016771246 podStartE2EDuration="2.016771246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:12.014162829 +0000 UTC m=+1934.011078213" watchObservedRunningTime="2026-10-30 19:39:12.016771246 +0000 UTC m=+1934.013686629"
Oct 30 19:39:12.852999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:12.852962    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:13.225182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.225148    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:13.386816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.386764    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-xks-ca\") pod \"836fdc23-6f7d-4a60-b829-d2969da31873\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") "
Oct 30 19:39:13.387085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.386834    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836fdc23-6f7d-4a60-b829-d2969da31873-kube-api-access-zvr7b\" (UniqueName: \"kubernetes.io/projected/836fdc23-6f7d-4a60-b829-d2969da31873-kube-api-access-zvr7b\") pod \"836fdc23-6f7d-4a60-b829-d2969da31873\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") "
Oct 30 19:39:13.388154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.387164    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-openshift-ca\") pod \"836fdc23-6f7d-4a60-b829-d2969da31873\" (UID: \"836fdc23-6f7d-4a60-b829-d2969da31873\") "
Oct 30 19:39:13.388154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.387536    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-openshift-ca" pod "836fdc23-6f7d-4a60-b829-d2969da31873" (UID: "836fdc23-6f7d-4a60-b829-d2969da31873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:13.388444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.388417    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:13.389224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.389190    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-xks-ca" pod "836fdc23-6f7d-4a60-b829-d2969da31873" (UID: "836fdc23-6f7d-4a60-b829-d2969da31873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:13.390182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.389272    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836fdc23-6f7d-4a60-b829-d2969da31873-kube-api-access-zvr7b" pod "836fdc23-6f7d-4a60-b829-d2969da31873" (UID: "836fdc23-6f7d-4a60-b829-d2969da31873"). InnerVolumeSpecName "kube-api-access-zvr7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:13.492167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.491942    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836fdc23-6f7d-4a60-b829-d2969da31873-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:13.492167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:13.491982    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvr7b\" (UniqueName: \"kubernetes.io/projected/836fdc23-6f7d-4a60-b829-d2969da31873-kube-api-access-zvr7b\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:13.677226 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:39:13.677181    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:39:14.014803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:14.014773    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"
Oct 30 19:39:14.015268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:14.014809    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh" event={"ID":"836fdc23-6f7d-4a60-b829-d2969da31873","Type":"ContainerDied","Data":"ff949cb11cc8251e33c547cf8c75d55e60465a06599e8a00a0cd79f79dcfd0e1"}
Oct 30 19:39:14.015268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:14.014858    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff949cb11cc8251e33c547cf8c75d55e60465a06599e8a00a0cd79f79dcfd0e1"
Oct 30 19:39:18.311084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.300401    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"]
Oct 30 19:39:18.311084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.309277    6777 cpu_manager.go:414] "containerMap: removing container" podUID="836fdc23-6f7d-4a60-b829-d2969da31873" containerName="revoke-keys"
Oct 30 19:39:18.311084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.309310    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836fdc23-6f7d-4a60-b829-d2969da31873" containerName="revoke-keys"
Oct 30 19:39:18.311084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.309570    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="836fdc23-6f7d-4a60-b829-d2969da31873" containerName="revoke-keys"
Oct 30 19:39:18.367113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.363528    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"]
Oct 30 19:39:18.367113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.363672    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.372134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.372001    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:39:18.372134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.372066    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:18.444410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.444360    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrv8\" (UniqueName: \"kubernetes.io/projected/86f95fb5-853d-4a6d-a847-9833f229b146-kube-api-access-cwrv8\") pod \"maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.444603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.444580    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.444743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.444717    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.547331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.547285    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.547532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.547385    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.547613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.547598    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrv8\" (UniqueName: \"kubernetes.io/projected/86f95fb5-853d-4a6d-a847-9833f229b146-kube-api-access-cwrv8\") pod \"maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.547973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.547949    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.550026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.549999    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.561327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.561251    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrv8\" (UniqueName: \"kubernetes.io/projected/86f95fb5-853d-4a6d-a847-9833f229b146-kube-api-access-cwrv8\") pod \"maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.691392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.691343    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:18.906517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.906427    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main" containerID="cri-o://f867b903d8ceaa9d7c461307c0a622a9a30559420ea4998333e2c59a4b489234" gracePeriod=45
Oct 30 19:39:18.987782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.987743    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"]
Oct 30 19:39:18.988675 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:39:18.988647    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f95fb5_853d_4a6d_a847_9833f229b146.slice/crio-d1ca250527d13111cbae983b95e860e7f22d4dcba71f96f8363c47dfe4806803 WatchSource:0}: Error finding container d1ca250527d13111cbae983b95e860e7f22d4dcba71f96f8363c47dfe4806803: Status 404 returned error can't find the container with id d1ca250527d13111cbae983b95e860e7f22d4dcba71f96f8363c47dfe4806803
Oct 30 19:39:18.991195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:18.991159    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/ready\": dial tcp 10.134.0.195:8000: connect: connection refused"
Oct 30 19:39:19.119098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.102952    6777 generic.go:356] "Generic (PLEG): container finished" podID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerID="f867b903d8ceaa9d7c461307c0a622a9a30559420ea4998333e2c59a4b489234" exitCode=0
Oct 30 19:39:19.119098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.103305    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" event={"ID":"aff4e94e-4e6a-463e-84a4-60ae69538dd9","Type":"ContainerDied","Data":"f867b903d8ceaa9d7c461307c0a622a9a30559420ea4998333e2c59a4b489234"}
Oct 30 19:39:19.119098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.105769    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976" event={"ID":"86f95fb5-853d-4a6d-a847-9833f229b146","Type":"ContainerStarted","Data":"d1ca250527d13111cbae983b95e860e7f22d4dcba71f96f8363c47dfe4806803"}
Oct 30 19:39:19.278975 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.278932    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:39:19.364967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.364901    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tmp-dir\") pod \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") "
Oct 30 19:39:19.365425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.365214    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tmp-dir" pod "aff4e94e-4e6a-463e-84a4-60ae69538dd9" (UID: "aff4e94e-4e6a-463e-84a4-60ae69538dd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:19.365425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.365235    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tls-certs\") pod \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") "
Oct 30 19:39:19.365425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.365323    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff4e94e-4e6a-463e-84a4-60ae69538dd9-kube-api-access-qcndl\" (UniqueName: \"kubernetes.io/projected/aff4e94e-4e6a-463e-84a4-60ae69538dd9-kube-api-access-qcndl\") pod \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") "
Oct 30 19:39:19.365425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.365402    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-home\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-home\") pod \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") "
Oct 30 19:39:19.365674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.365443    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-dshm\") pod \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") "
Oct 30 19:39:19.365674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.365474    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-model-cache\") pod \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\" (UID: \"aff4e94e-4e6a-463e-84a4-60ae69538dd9\") "
Oct 30 19:39:19.365802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.365684    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-home" pod "aff4e94e-4e6a-463e-84a4-60ae69538dd9" (UID: "aff4e94e-4e6a-463e-84a4-60ae69538dd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:19.366387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.366359    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-model-cache" pod "aff4e94e-4e6a-463e-84a4-60ae69538dd9" (UID: "aff4e94e-4e6a-463e-84a4-60ae69538dd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:19.367202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.366747    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:19.367202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.366772    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:19.367202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.366800    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:19.368361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.368335    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-dshm" pod "aff4e94e-4e6a-463e-84a4-60ae69538dd9" (UID: "aff4e94e-4e6a-463e-84a4-60ae69538dd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:19.368448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.368335    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tls-certs" pod "aff4e94e-4e6a-463e-84a4-60ae69538dd9" (UID: "aff4e94e-4e6a-463e-84a4-60ae69538dd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:19.368501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.368434    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff4e94e-4e6a-463e-84a4-60ae69538dd9-kube-api-access-qcndl" pod "aff4e94e-4e6a-463e-84a4-60ae69538dd9" (UID: "aff4e94e-4e6a-463e-84a4-60ae69538dd9"). InnerVolumeSpecName "kube-api-access-qcndl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:19.469148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.469112    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff4e94e-4e6a-463e-84a4-60ae69538dd9-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:19.469148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.469153    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcndl\" (UniqueName: \"kubernetes.io/projected/aff4e94e-4e6a-463e-84a4-60ae69538dd9-kube-api-access-qcndl\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:19.469393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:19.469171    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff4e94e-4e6a-463e-84a4-60ae69538dd9-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:20.130952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:20.130918    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"
Oct 30 19:39:20.135704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:20.135664    6777 generic.go:356] "Generic (PLEG): container finished" podID="86f95fb5-853d-4a6d-a847-9833f229b146" containerID="4f4932abe864d1b1a7b574ce4cc0c2e38231f884f08c5b442345838d83e2bb71" exitCode=0
Oct 30 19:39:20.138856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:20.138812    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79" event={"ID":"aff4e94e-4e6a-463e-84a4-60ae69538dd9","Type":"ContainerDied","Data":"0fd82f5dc0480dbf995df9dba95800608cceba89a056621801a312d85a405122"}
Oct 30 19:39:20.139039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:20.138882    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976" event={"ID":"86f95fb5-853d-4a6d-a847-9833f229b146","Type":"ContainerDied","Data":"4f4932abe864d1b1a7b574ce4cc0c2e38231f884f08c5b442345838d83e2bb71"}
Oct 30 19:39:20.139039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:20.138929    6777 scope.go:122] "RemoveContainer" containerID="f867b903d8ceaa9d7c461307c0a622a9a30559420ea4998333e2c59a4b489234"
Oct 30 19:39:20.175468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:20.175429    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"]
Oct 30 19:39:20.180515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:20.180478    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f2dbd9/test-model-f2dbd9-kserve-5876c4484f-btw79"]
Oct 30 19:39:20.853758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:20.853722    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:20.867258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:20.867225    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:21.160841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.160801    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:21.392505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.392473    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:21.532903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.530706    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-openshift-ca\") pod \"86f95fb5-853d-4a6d-a847-9833f229b146\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") "
Oct 30 19:39:21.532903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.530838    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-xks-ca\") pod \"86f95fb5-853d-4a6d-a847-9833f229b146\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") "
Oct 30 19:39:21.532903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.531201    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f95fb5-853d-4a6d-a847-9833f229b146-kube-api-access-cwrv8\" (UniqueName: \"kubernetes.io/projected/86f95fb5-853d-4a6d-a847-9833f229b146-kube-api-access-cwrv8\") pod \"86f95fb5-853d-4a6d-a847-9833f229b146\" (UID: \"86f95fb5-853d-4a6d-a847-9833f229b146\") "
Oct 30 19:39:21.534122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.532917    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-openshift-ca" pod "86f95fb5-853d-4a6d-a847-9833f229b146" (UID: "86f95fb5-853d-4a6d-a847-9833f229b146"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:21.546249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.546205    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f95fb5-853d-4a6d-a847-9833f229b146-kube-api-access-cwrv8" pod "86f95fb5-853d-4a6d-a847-9833f229b146" (UID: "86f95fb5-853d-4a6d-a847-9833f229b146"). InnerVolumeSpecName "kube-api-access-cwrv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:21.547812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.547770    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-xks-ca" pod "86f95fb5-853d-4a6d-a847-9833f229b146" (UID: "86f95fb5-853d-4a6d-a847-9833f229b146"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:21.633783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.633738    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwrv8\" (UniqueName: \"kubernetes.io/projected/86f95fb5-853d-4a6d-a847-9833f229b146-kube-api-access-cwrv8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:21.633783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.633773    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:21.634021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:21.633800    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f95fb5-853d-4a6d-a847-9833f229b146-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:22.118525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:22.118489    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" path="/var/lib/kubelet/pods/aff4e94e-4e6a-463e-84a4-60ae69538dd9/volumes"
Oct 30 19:39:22.160813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:22.160778    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"
Oct 30 19:39:22.160813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:22.160801    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976" event={"ID":"86f95fb5-853d-4a6d-a847-9833f229b146","Type":"ContainerDied","Data":"d1ca250527d13111cbae983b95e860e7f22d4dcba71f96f8363c47dfe4806803"}
Oct 30 19:39:22.161048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:22.160859    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1ca250527d13111cbae983b95e860e7f22d4dcba71f96f8363c47dfe4806803"
Oct 30 19:39:23.035226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:23.035187    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"]
Oct 30 19:39:23.041658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:23.040654    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6a7368-cmqbs"]
Oct 30 19:39:24.116625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:24.116592    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9239a6b1-236b-4e72-a8ae-fabc64d22619" path="/var/lib/kubelet/pods/9239a6b1-236b-4e72-a8ae-fabc64d22619/volumes"
Oct 30 19:39:33.920085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:33.915415    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"]
Oct 30 19:39:33.943083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:33.941762    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f2dbd9-t4976"]
Oct 30 19:39:34.122378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:34.122342    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f95fb5-853d-4a6d-a847-9833f229b146" path="/var/lib/kubelet/pods/86f95fb5-853d-4a6d-a847-9833f229b146/volumes"
Oct 30 19:39:35.029268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:35.029223    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"]
Oct 30 19:39:35.034614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:35.034575    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b781b-mnmdh"]
Oct 30 19:39:36.227120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:36.227075    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f06436-7a0b-4124-9a73-7430a0d83fcb" path="/var/lib/kubelet/pods/94f06436-7a0b-4124-9a73-7430a0d83fcb/volumes"
Oct 30 19:39:43.120105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.068730    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.120105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.078165    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.120105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.086212    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.120105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.098333    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.120105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.098662    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Oct 30 19:39:43.120105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.098719    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.199147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.174647    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.286082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.199459    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.286082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.201178    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.286082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.201216    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.286082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.201965    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"]
Oct 30 19:39:43.286082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.285126    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy" containerID="cri-o://4c0cf75fc8bf0861914344452415199463b7811f7d4d4ac21e3a16dc1b273ef5" gracePeriod=30
Oct 30 19:39:43.587085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.584201    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Oct 30 19:39:43.708501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.681096    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.746966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:43.730677    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.001652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.001542    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.064086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.043223    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.130085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.124696    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.179086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.152941    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:39:44.183241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.181164    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.183241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.181599    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.183241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.181903    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.183241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.182179    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.303153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.300950    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.303880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.303827    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.333996 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.308616    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.380411    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.392236    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.767110    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.767148    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:39:44.819283    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.926023    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966351    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966404    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966438    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966470    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966504    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966534    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966567    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966602    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966633    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:44.966666    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.053191    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.140024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.138767    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:39:45.144194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.144152    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.204092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.200323    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.304087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.230048    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.398147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.241486    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.503087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.241430    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.560347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.314343    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.568270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.501134    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.616212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.616173    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:39:45.984270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.961778    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.984270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.969177    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.984270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.973712    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.984270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:45.974035    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.145156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.145023    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.173293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.173242    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.173868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.148118    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.524233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.522218    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.551792 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.542788    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.589084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.552151    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.589084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.552488    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.655139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.590003    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.655139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.598383    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.740548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.731229    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:39:46.740870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.740833    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.915418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.915383    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:39:46.965294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:46.965239    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.069231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.068240    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.086421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.051008    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:39:47.088409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.088375    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:39:47.101136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.090548    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:39:47.125022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.118319    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:39:47.141379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.141345    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:39:47.141740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.141723    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:39:47.141866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.141856    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:39:47.144293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.144208    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:39:47.187142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.174470    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:39:47.202998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.202951    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:39:47.208707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.208666    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:39:47.212644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:47.212613    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:39:49.102689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.102655    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"]
Oct 30 19:39:49.141939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.103698    6777 cpu_manager.go:414] "containerMap: removing container" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main"
Oct 30 19:39:49.141939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.103721    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main"
Oct 30 19:39:49.141939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.103747    6777 cpu_manager.go:414] "containerMap: removing container" podUID="86f95fb5-853d-4a6d-a847-9833f229b146" containerName="revoke-keys"
Oct 30 19:39:49.141939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.103752    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f95fb5-853d-4a6d-a847-9833f229b146" containerName="revoke-keys"
Oct 30 19:39:49.141939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.103924    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff4e94e-4e6a-463e-84a4-60ae69538dd9" containerName="main"
Oct 30 19:39:49.141939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.103937    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f95fb5-853d-4a6d-a847-9833f229b146" containerName="revoke-keys"
Oct 30 19:39:49.404752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.404649    6777 generic.go:356] "Generic (PLEG): container finished" podID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerID="4c0cf75fc8bf0861914344452415199463b7811f7d4d4ac21e3a16dc1b273ef5" exitCode=0
Oct 30 19:39:49.664733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.664634    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" event={"ID":"c13df2c0-4690-4548-bf78-a4205a6cae13","Type":"ContainerDied","Data":"4c0cf75fc8bf0861914344452415199463b7811f7d4d4ac21e3a16dc1b273ef5"}
Oct 30 19:39:49.664733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.664711    6777 scope.go:122] "RemoveContainer" containerID="2fdc81f7cc88d01643eda59c2118cff4ba5f8101223e33927ec67bc91ddfc760"
Oct 30 19:39:49.664942 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.664792    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:49.839399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.743473    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e273b0d-lh4hc\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:49.839399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.743874    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvv27\" (UniqueName: \"kubernetes.io/projected/7ffca28b-019b-454e-9681-81fc0de7c912-kube-api-access-pvv27\") pod \"maas-api-revoke-sub-0e273b0d-lh4hc\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:49.839399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.743992    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e273b0d-lh4hc\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:49.846026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.845994    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvv27\" (UniqueName: \"kubernetes.io/projected/7ffca28b-019b-454e-9681-81fc0de7c912-kube-api-access-pvv27\") pod \"maas-api-revoke-sub-0e273b0d-lh4hc\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:49.867351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.846171    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e273b0d-lh4hc\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:49.867351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.846320    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e273b0d-lh4hc\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:49.867351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.860121    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:49.867351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.860121    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:39:49.870218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:49.870183    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e273b0d-lh4hc\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:50.339251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:50.339218    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e273b0d-lh4hc\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:50.339643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:50.339549    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvv27\" (UniqueName: \"kubernetes.io/projected/7ffca28b-019b-454e-9681-81fc0de7c912-kube-api-access-pvv27\") pod \"maas-api-revoke-sub-0e273b0d-lh4hc\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:50.438704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:50.438665    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:51.091049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:50.982464    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"]
Oct 30 19:39:51.189609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.189571    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:39:51.236591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.236551    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"]
Oct 30 19:39:51.238515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.238489    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy"
Oct 30 19:39:51.238794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.238761    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy"
Oct 30 19:39:51.238994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.238971    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy"
Oct 30 19:39:51.239139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.239124    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy"
Oct 30 19:39:51.240804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.239542    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy"
Oct 30 19:39:51.240804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.239572    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" containerName="istio-proxy"
Oct 30 19:39:51.253146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.252107    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.258588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.258199    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"]
Oct 30 19:39:51.258588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.258544    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-1b1531-openshift-default-dockercfg-5ppz9\""
Oct 30 19:39:51.293457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.293361    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13df2c0-4690-4548-bf78-a4205a6cae13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13df2c0-4690-4548-bf78-a4205a6cae13-istiod-ca-cert\") pod \"c13df2c0-4690-4548-bf78-a4205a6cae13\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") "
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.293516    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-certs\") pod \"c13df2c0-4690-4548-bf78-a4205a6cae13\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") "
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.293702    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-credential-socket\") pod \"c13df2c0-4690-4548-bf78-a4205a6cae13\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") "
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.293760    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13df2c0-4690-4548-bf78-a4205a6cae13-istiod-ca-cert" pod "c13df2c0-4690-4548-bf78-a4205a6cae13" (UID: "c13df2c0-4690-4548-bf78-a4205a6cae13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.293867    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-envoy\") pod \"c13df2c0-4690-4548-bf78-a4205a6cae13\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") "
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.293979    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-podinfo\") pod \"c13df2c0-4690-4548-bf78-a4205a6cae13\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") "
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.294014    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-token\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-token\") pod \"c13df2c0-4690-4548-bf78-a4205a6cae13\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") "
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.294026    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-credential-socket" pod "c13df2c0-4690-4548-bf78-a4205a6cae13" (UID: "c13df2c0-4690-4548-bf78-a4205a6cae13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.294095    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-data\") pod \"c13df2c0-4690-4548-bf78-a4205a6cae13\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") "
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.294158    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-socket\") pod \"c13df2c0-4690-4548-bf78-a4205a6cae13\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") "
Oct 30 19:39:51.294363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.294204    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-kube-api-access-2xfzx\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-kube-api-access-2xfzx\") pod \"c13df2c0-4690-4548-bf78-a4205a6cae13\" (UID: \"c13df2c0-4690-4548-bf78-a4205a6cae13\") "
Oct 30 19:39:51.294879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.294486    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-data" pod "c13df2c0-4690-4548-bf78-a4205a6cae13" (UID: "c13df2c0-4690-4548-bf78-a4205a6cae13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:51.294879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.294719    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-socket" pod "c13df2c0-4690-4548-bf78-a4205a6cae13" (UID: "c13df2c0-4690-4548-bf78-a4205a6cae13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:51.294879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.294778    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-certs" pod "c13df2c0-4690-4548-bf78-a4205a6cae13" (UID: "c13df2c0-4690-4548-bf78-a4205a6cae13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:51.296334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.295293    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.296334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.295320    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.296334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.295335    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.296334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.295350    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.296334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.295365    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13df2c0-4690-4548-bf78-a4205a6cae13-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.297675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.297651    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-envoy" pod "c13df2c0-4690-4548-bf78-a4205a6cae13" (UID: "c13df2c0-4690-4548-bf78-a4205a6cae13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:51.298124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.298099    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-podinfo" pod "c13df2c0-4690-4548-bf78-a4205a6cae13" (UID: "c13df2c0-4690-4548-bf78-a4205a6cae13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:51.298914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.298134    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-token" pod "c13df2c0-4690-4548-bf78-a4205a6cae13" (UID: "c13df2c0-4690-4548-bf78-a4205a6cae13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:51.300546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.300523    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-kube-api-access-2xfzx" pod "c13df2c0-4690-4548-bf78-a4205a6cae13" (UID: "c13df2c0-4690-4548-bf78-a4205a6cae13"). InnerVolumeSpecName "kube-api-access-2xfzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:51.396639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.396593    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.396639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.396642    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da534d-2d99-4e05-8866-6915937d2c14-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.397100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.396837    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.397100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.396942    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da534d-2d99-4e05-8866-6915937d2c14-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.397100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.397000    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.397100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.397018    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.397241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.397176    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.397311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.397296    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpmb\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-kube-api-access-xlpmb\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.397377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.397364    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.397528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.397514    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.397564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.397532    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.397564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.397541    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.397564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.397551    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xfzx\" (UniqueName: \"kubernetes.io/projected/c13df2c0-4690-4548-bf78-a4205a6cae13-kube-api-access-2xfzx\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.451554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.451519    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"]
Oct 30 19:39:51.482961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.482931    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"
Oct 30 19:39:51.482961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.482942    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d" event={"ID":"c13df2c0-4690-4548-bf78-a4205a6cae13","Type":"ContainerDied","Data":"9fb8ceecf5145e81b489b41811b308da97ce9fada11b8abddc7358e300272d7f"}
Oct 30 19:39:51.483368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.483040    6777 scope.go:122] "RemoveContainer" containerID="4c0cf75fc8bf0861914344452415199463b7811f7d4d4ac21e3a16dc1b273ef5"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.498614    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpmb\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-kube-api-access-xlpmb\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.498702    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.498758    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.498778    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da534d-2d99-4e05-8866-6915937d2c14-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.498914    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.498972    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da534d-2d99-4e05-8866-6915937d2c14-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.499041    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.499071    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.499178    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.499283    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.499327    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.499355    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.499554    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.499667    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da534d-2d99-4e05-8866-6915937d2c14-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.501306    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.502531    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da534d-2d99-4e05-8866-6915937d2c14-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.513258    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpmb\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-kube-api-access-xlpmb\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.518157    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"]
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.520712    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.555726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.525921    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-8wp8d"]
Oct 30 19:39:51.574625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.574592    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:51.918249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.918214    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"]
Oct 30 19:39:51.921594 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:39:51.921560    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9da534d_2d99_4e05_8866_6915937d2c14.slice/crio-40fe429c0f159e53056bdd38df5a478d590f97b4d152d7b90a48d9c1fb473b4f WatchSource:0}: Error finding container 40fe429c0f159e53056bdd38df5a478d590f97b4d152d7b90a48d9c1fb473b4f: Status 404 returned error can't find the container with id 40fe429c0f159e53056bdd38df5a478d590f97b4d152d7b90a48d9c1fb473b4f
Oct 30 19:39:51.925404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.925368    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:39:51.925604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.925445    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:39:51.925604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:51.925476    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:39:52.083129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.083093    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"]
Oct 30 19:39:52.083352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.083315    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy" containerID="cri-o://c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9" gracePeriod=30
Oct 30 19:39:52.132336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.132231    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13df2c0-4690-4548-bf78-a4205a6cae13" path="/var/lib/kubelet/pods/c13df2c0-4690-4548-bf78-a4205a6cae13/volumes"
Oct 30 19:39:52.495895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.495855    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" event={"ID":"b9da534d-2d99-4e05-8866-6915937d2c14","Type":"ContainerStarted","Data":"16a2eaeaf9f8b4f67c93b39e5c09039e0ce85be048033c3a550ef923962b2a8c"}
Oct 30 19:39:52.496381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.495915    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" event={"ID":"b9da534d-2d99-4e05-8866-6915937d2c14","Type":"ContainerStarted","Data":"40fe429c0f159e53056bdd38df5a478d590f97b4d152d7b90a48d9c1fb473b4f"}
Oct 30 19:39:52.497781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.497749    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc" event={"ID":"7ffca28b-019b-454e-9681-81fc0de7c912","Type":"ContainerStarted","Data":"cc22aa89829045f5d382f6c5705243bee7662f4d18d786d2c4c5778da7318093"}
Oct 30 19:39:52.497907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.497803    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc" event={"ID":"7ffca28b-019b-454e-9681-81fc0de7c912","Type":"ContainerStarted","Data":"b7e0351c442a853a36f334c76543ff9f78ebbfb314e4268c9d8d332a58d5e694"}
Oct 30 19:39:52.525202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.525139    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" podStartSLOduration=52.525125554 podStartE2EDuration="52.525125554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:52.524560702 +0000 UTC m=+1974.521476086" watchObservedRunningTime="2026-10-30 19:39:52.525125554 +0000 UTC m=+1974.522040939"
Oct 30 19:39:52.547946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.547885    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc" podStartSLOduration=3.547868593 podStartE2EDuration="3.547868593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:52.545451032 +0000 UTC m=+1974.542366426" watchObservedRunningTime="2026-10-30 19:39:52.547868593 +0000 UTC m=+1974.544783977"
Oct 30 19:39:52.574875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:52.574832    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:53.583609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:53.583549    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:39:53.600439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:53.600325    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 19:39:53.605644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:53.605608    6777 generic.go:356] "Generic (PLEG): container finished" podID="7ffca28b-019b-454e-9681-81fc0de7c912" containerID="cc22aa89829045f5d382f6c5705243bee7662f4d18d786d2c4c5778da7318093" exitCode=0
Oct 30 19:39:53.605817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:53.605773    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc" event={"ID":"7ffca28b-019b-454e-9681-81fc0de7c912","Type":"ContainerDied","Data":"cc22aa89829045f5d382f6c5705243bee7662f4d18d786d2c4c5778da7318093"}
Oct 30 19:39:54.033768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:54.033728    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"]
Oct 30 19:39:54.077091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:54.048656    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"]
Oct 30 19:39:54.584085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:54.575507    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:39:54.584085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:54.575621    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 19:39:55.032394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.032365    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:55.089555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.089519    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ffca28b-019b-454e-9681-81fc0de7c912-kube-api-access-pvv27\" (UniqueName: \"kubernetes.io/projected/7ffca28b-019b-454e-9681-81fc0de7c912-kube-api-access-pvv27\") pod \"7ffca28b-019b-454e-9681-81fc0de7c912\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") "
Oct 30 19:39:55.089738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.089670    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-openshift-ca\") pod \"7ffca28b-019b-454e-9681-81fc0de7c912\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") "
Oct 30 19:39:55.089956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.089933    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-xks-ca\") pod \"7ffca28b-019b-454e-9681-81fc0de7c912\" (UID: \"7ffca28b-019b-454e-9681-81fc0de7c912\") "
Oct 30 19:39:55.090118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.090052    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-openshift-ca" pod "7ffca28b-019b-454e-9681-81fc0de7c912" (UID: "7ffca28b-019b-454e-9681-81fc0de7c912"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:55.090773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.090750    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:55.092491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.092465    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-xks-ca" pod "7ffca28b-019b-454e-9681-81fc0de7c912" (UID: "7ffca28b-019b-454e-9681-81fc0de7c912"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:55.092708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.092532    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffca28b-019b-454e-9681-81fc0de7c912-kube-api-access-pvv27" pod "7ffca28b-019b-454e-9681-81fc0de7c912" (UID: "7ffca28b-019b-454e-9681-81fc0de7c912"). InnerVolumeSpecName "kube-api-access-pvv27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:55.193032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.192987    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffca28b-019b-454e-9681-81fc0de7c912-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:55.193032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.193034    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvv27\" (UniqueName: \"kubernetes.io/projected/7ffca28b-019b-454e-9681-81fc0de7c912-kube-api-access-pvv27\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:55.597080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.585161    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:39:55.597080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.585240    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 19:39:55.844126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.839752    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"
Oct 30 19:39:55.844126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.839814    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc" event={"ID":"7ffca28b-019b-454e-9681-81fc0de7c912","Type":"ContainerDied","Data":"b7e0351c442a853a36f334c76543ff9f78ebbfb314e4268c9d8d332a58d5e694"}
Oct 30 19:39:55.844126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:55.839867    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e0351c442a853a36f334c76543ff9f78ebbfb314e4268c9d8d332a58d5e694"
Oct 30 19:39:56.575535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:56.575491    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:39:56.575752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:56.575573    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 19:39:57.575428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.575387    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:39:57.576400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.576362    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 19:39:57.880158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.880127    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:39:57.884914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.884881    6777 generic.go:356] "Generic (PLEG): container finished" podID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerID="c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9" exitCode=0
Oct 30 19:39:57.885100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.884927    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"
Oct 30 19:39:57.885100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.884959    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" event={"ID":"bad79567-24bd-4ed5-9be8-59c3a09b9e6d","Type":"ContainerDied","Data":"c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9"}
Oct 30 19:39:57.885100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.885019    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx" event={"ID":"bad79567-24bd-4ed5-9be8-59c3a09b9e6d","Type":"ContainerDied","Data":"a36dfa45fb226b2d5a1d26fd4bc003c3e0d34c8309370cbf3f9ffe2d06279fe7"}
Oct 30 19:39:57.886111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.885045    6777 scope.go:122] "RemoveContainer" containerID="c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9"
Oct 30 19:39:57.897720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.897696    6777 scope.go:122] "RemoveContainer" containerID="c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9"
Oct 30 19:39:57.898095 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:39:57.898070    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9\": container with ID starting with c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9 not found: ID does not exist" containerID="c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9"
Oct 30 19:39:57.899009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.898105    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9"} err="failed to get container status \"c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9\": rpc error: code = NotFound desc = could not find container \"c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9\": container with ID starting with c540fdfada6040445a0568f156e75617cc8c51e81d4b785273b61ab3694bffd9 not found: ID does not exist"
Oct 30 19:39:57.933391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.933348    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"]
Oct 30 19:39:57.934367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.934344    6777 cpu_manager.go:414] "containerMap: removing container" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy"
Oct 30 19:39:57.934367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.934365    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy"
Oct 30 19:39:57.934574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.934392    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7ffca28b-019b-454e-9681-81fc0de7c912" containerName="revoke-keys"
Oct 30 19:39:57.934574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.934398    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ffca28b-019b-454e-9681-81fc0de7c912" containerName="revoke-keys"
Oct 30 19:39:57.934574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.934570    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" containerName="istio-proxy"
Oct 30 19:39:57.934736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.934585    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ffca28b-019b-454e-9681-81fc0de7c912" containerName="revoke-keys"
Oct 30 19:39:57.939882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.939855    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:57.943281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.943249    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-credential-socket\") pod \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") "
Oct 30 19:39:57.943438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.943308    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-podinfo\") pod \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") "
Oct 30 19:39:57.943501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.943461    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istiod-ca-cert\") pod \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") "
Oct 30 19:39:57.943588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.943564    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-token\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-token\") pod \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") "
Oct 30 19:39:57.943588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.943570    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-credential-socket" pod "bad79567-24bd-4ed5-9be8-59c3a09b9e6d" (UID: "bad79567-24bd-4ed5-9be8-59c3a09b9e6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:57.943791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.943766    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-certs\") pod \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") "
Oct 30 19:39:57.943898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.943873    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istiod-ca-cert" pod "bad79567-24bd-4ed5-9be8-59c3a09b9e6d" (UID: "bad79567-24bd-4ed5-9be8-59c3a09b9e6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:57.943999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.943981    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-socket\") pod \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") "
Oct 30 19:39:57.944144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.944126    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-data\") pod \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") "
Oct 30 19:39:57.944290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.944276    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-envoy\") pod \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") "
Oct 30 19:39:57.944695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.944405    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-socket" pod "bad79567-24bd-4ed5-9be8-59c3a09b9e6d" (UID: "bad79567-24bd-4ed5-9be8-59c3a09b9e6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:57.944695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.944527    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-kube-api-access-mwq8m\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-kube-api-access-mwq8m\") pod \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\" (UID: \"bad79567-24bd-4ed5-9be8-59c3a09b9e6d\") "
Oct 30 19:39:57.944695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.944569    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-certs" pod "bad79567-24bd-4ed5-9be8-59c3a09b9e6d" (UID: "bad79567-24bd-4ed5-9be8-59c3a09b9e6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:57.944695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.944663    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-data" pod "bad79567-24bd-4ed5-9be8-59c3a09b9e6d" (UID: "bad79567-24bd-4ed5-9be8-59c3a09b9e6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:57.945700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.945674    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:57.946560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.945704    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:57.946560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.945718    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:57.946560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.945731    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:57.946560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.945745    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:57.947958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.947880    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-kube-api-access-mwq8m" pod "bad79567-24bd-4ed5-9be8-59c3a09b9e6d" (UID: "bad79567-24bd-4ed5-9be8-59c3a09b9e6d"). InnerVolumeSpecName "kube-api-access-mwq8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:57.947958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.947921    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-token" pod "bad79567-24bd-4ed5-9be8-59c3a09b9e6d" (UID: "bad79567-24bd-4ed5-9be8-59c3a09b9e6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:57.969765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.969697    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-envoy" pod "bad79567-24bd-4ed5-9be8-59c3a09b9e6d" (UID: "bad79567-24bd-4ed5-9be8-59c3a09b9e6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:57.996809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:57.996531    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"]
Oct 30 19:39:58.012649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.009689    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-podinfo" pod "bad79567-24bd-4ed5-9be8-59c3a09b9e6d" (UID: "bad79567-24bd-4ed5-9be8-59c3a09b9e6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:58.047454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.047385    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53b1878-fc90-406e-bfcc-ac85764c7cec-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.047659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.047580    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8qm\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-kube-api-access-6b8qm\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.047764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.047692    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.047764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.047751    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.047930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.047900    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.048047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.048028    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.048636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.048159    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.048636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.048232    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.048636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.048277    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.048636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.048433    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:58.048636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.048454    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:58.048636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.048471    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:58.048636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.048486    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwq8m\" (UniqueName: \"kubernetes.io/projected/bad79567-24bd-4ed5-9be8-59c3a09b9e6d-kube-api-access-mwq8m\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:58.231890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.231847    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.232084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.231942    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.232084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.231982    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.232238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.232168    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53b1878-fc90-406e-bfcc-ac85764c7cec-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.232445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.232301    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8qm\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-kube-api-access-6b8qm\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.232445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.232345    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.232445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.232379    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.236300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.232451    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.236300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.232520    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.236300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.232927    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.236300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.233202    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.236300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.233545    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.236300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.234177    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.236300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.235284    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53b1878-fc90-406e-bfcc-ac85764c7cec-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.304686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.283265    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.304686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.290158    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.336381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.307326    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8qm\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-kube-api-access-6b8qm\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.336381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.332203    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"]
Oct 30 19:39:58.336381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.334109    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.336381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.335493    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-zlrjx"]
Oct 30 19:39:58.576091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.575933    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:39:58.576091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.576000    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 19:39:58.621847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.621811    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:39:58.946841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.946808    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"]
Oct 30 19:39:58.946974 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:39:58.946852    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53b1878_fc90_406e_bfcc_ac85764c7cec.slice/crio-37acdab2f828a9d4cd134598d614f3381d724e42ceb273be7d074f538470b62b WatchSource:0}: Error finding container 37acdab2f828a9d4cd134598d614f3381d724e42ceb273be7d074f538470b62b: Status 404 returned error can't find the container with id 37acdab2f828a9d4cd134598d614f3381d724e42ceb273be7d074f538470b62b
Oct 30 19:39:58.968569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.968521    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:39:58.969916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.968609    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:39:58.969916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:58.968648    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:39:59.086090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.077792    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp" podUID="87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" containerName="payload-pre-processing" containerID="cri-o://5ebc2f7b4f64e0718dcc4b4a524702cee1f0cd0c438c535376376569032d348d" gracePeriod=25
Oct 30 19:39:59.086090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.082233    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f" podUID="cb9152f3-269a-46bc-b268-7a8173084433" containerName="payload-processing" containerID="cri-o://43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21" gracePeriod=25
Oct 30 19:39:59.710247 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.710193    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"]
Oct 30 19:39:59.716715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.715881    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.721179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.720893    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:59.721485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.721452    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:59.721743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.721725    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:39:59.732165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.722610    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:59.732165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.723118    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"]
Oct 30 19:39:59.732165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.724403    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:39:59.793426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.793390    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5xbp\" (UniqueName: \"kubernetes.io/projected/84556a7c-211f-48e3-9211-ebc8bc5b79d5-kube-api-access-k5xbp\") pod \"maas-api-revoke-sub-08a49217-f6xn9\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.793609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.793475    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08a49217-f6xn9\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.793897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.793868    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08a49217-f6xn9\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.908134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.900805    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5xbp\" (UniqueName: \"kubernetes.io/projected/84556a7c-211f-48e3-9211-ebc8bc5b79d5-kube-api-access-k5xbp\") pod \"maas-api-revoke-sub-08a49217-f6xn9\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.908134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.900915    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08a49217-f6xn9\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.908134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.901251    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08a49217-f6xn9\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.914557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.914119    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08a49217-f6xn9\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.916542 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.915151    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08a49217-f6xn9\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.944050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.942337    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5xbp\" (UniqueName: \"kubernetes.io/projected/84556a7c-211f-48e3-9211-ebc8bc5b79d5-kube-api-access-k5xbp\") pod \"maas-api-revoke-sub-08a49217-f6xn9\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:39:59.944050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.942671    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" event={"ID":"c53b1878-fc90-406e-bfcc-ac85764c7cec","Type":"ContainerStarted","Data":"6adfd2d06f7a918124211ca69f6d52079608a98488f4d15f0a944956f54a0b0d"}
Oct 30 19:39:59.944050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.942737    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" event={"ID":"c53b1878-fc90-406e-bfcc-ac85764c7cec","Type":"ContainerStarted","Data":"37acdab2f828a9d4cd134598d614f3381d724e42ceb273be7d074f538470b62b"}
Oct 30 19:39:59.969842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:39:59.969721    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" podStartSLOduration=59.969708117 podStartE2EDuration="59.969708117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:59.969500999 +0000 UTC m=+1981.966416408" watchObservedRunningTime="2026-10-30 19:39:59.969708117 +0000 UTC m=+1981.966623501"
Oct 30 19:40:00.043269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:00.043231    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:40:00.119533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:00.119462    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad79567-24bd-4ed5-9be8-59c3a09b9e6d" path="/var/lib/kubelet/pods/bad79567-24bd-4ed5-9be8-59c3a09b9e6d/volumes"
Oct 30 19:40:00.384474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:00.384422    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"]
Oct 30 19:40:00.430855 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:40:00.430818    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84556a7c_211f_48e3_9211_ebc8bc5b79d5.slice/crio-82a55e1e02946a4ea53795221de3942f5a1657e8afcbc02603e75e1065f8e597 WatchSource:0}: Error finding container 82a55e1e02946a4ea53795221de3942f5a1657e8afcbc02603e75e1065f8e597: Status 404 returned error can't find the container with id 82a55e1e02946a4ea53795221de3942f5a1657e8afcbc02603e75e1065f8e597
Oct 30 19:40:00.634079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:00.631292    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:40:00.649480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:00.641785    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:40:00.649480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:00.641839    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 19:40:00.733349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:00.733292    6777 scope.go:122] "RemoveContainer" containerID="0e9665f82a6d16a059927973707991a0e19795d675c3826e4eba1d37bb648731"
Oct 30 19:40:00.905406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:00.898674    6777 scope.go:122] "RemoveContainer" containerID="0d650fe88e87a8d358e628a106b12119a7ed1d1f38d772a1f9075d36457d543d"
Oct 30 19:40:01.106101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:01.106043    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"]
Oct 30 19:40:01.115510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:01.114996    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2a3340-szs9j"]
Oct 30 19:40:01.116017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:01.115991    6777 scope.go:122] "RemoveContainer" containerID="ee37b735f2b9562912f497b41eace9a362e7155f37b4a4422186ca70d98c6984"
Oct 30 19:40:01.123097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:01.122967    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9" event={"ID":"84556a7c-211f-48e3-9211-ebc8bc5b79d5","Type":"ContainerStarted","Data":"82a55e1e02946a4ea53795221de3942f5a1657e8afcbc02603e75e1065f8e597"}
Oct 30 19:40:01.622998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:01.622960    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:40:01.623268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:01.623097    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 19:40:02.118315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:02.118219    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a" path="/var/lib/kubelet/pods/5cc4cfcb-6fe1-4c3d-9e99-5f935a8e7a0a/volumes"
Oct 30 19:40:02.134305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:02.134262    6777 generic.go:356] "Generic (PLEG): container finished" podID="84556a7c-211f-48e3-9211-ebc8bc5b79d5" containerID="9911ae24c93010e7e2c64798a8765241497a7d037feefd6d3fa6012c469d7878" exitCode=0
Oct 30 19:40:02.134511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:02.134371    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9" event={"ID":"84556a7c-211f-48e3-9211-ebc8bc5b79d5","Type":"ContainerDied","Data":"9911ae24c93010e7e2c64798a8765241497a7d037feefd6d3fa6012c469d7878"}
Oct 30 19:40:02.277746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:02.277650    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp" podUID="87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:40:02.278260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:02.278216    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f" podUID="cb9152f3-269a-46bc-b268-7a8173084433" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:40:02.623197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:02.623161    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:40:02.623395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:02.623222    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 19:40:03.444763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.444733    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:40:03.580076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.579799    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-openshift-ca\") pod \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") "
Oct 30 19:40:03.580285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.580152    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-xks-ca\") pod \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") "
Oct 30 19:40:03.580285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.580200    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-openshift-ca" pod "84556a7c-211f-48e3-9211-ebc8bc5b79d5" (UID: "84556a7c-211f-48e3-9211-ebc8bc5b79d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:03.580285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.580265    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84556a7c-211f-48e3-9211-ebc8bc5b79d5-kube-api-access-k5xbp\" (UniqueName: \"kubernetes.io/projected/84556a7c-211f-48e3-9211-ebc8bc5b79d5-kube-api-access-k5xbp\") pod \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\" (UID: \"84556a7c-211f-48e3-9211-ebc8bc5b79d5\") "
Oct 30 19:40:03.583937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.581213    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:03.583937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.583567    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-xks-ca" pod "84556a7c-211f-48e3-9211-ebc8bc5b79d5" (UID: "84556a7c-211f-48e3-9211-ebc8bc5b79d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:03.584716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.584689    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84556a7c-211f-48e3-9211-ebc8bc5b79d5-kube-api-access-k5xbp" pod "84556a7c-211f-48e3-9211-ebc8bc5b79d5" (UID: "84556a7c-211f-48e3-9211-ebc8bc5b79d5"). InnerVolumeSpecName "kube-api-access-k5xbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:03.622684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.622638    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:40:03.622877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.622703    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 19:40:03.684408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.684284    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5xbp\" (UniqueName: \"kubernetes.io/projected/84556a7c-211f-48e3-9211-ebc8bc5b79d5-kube-api-access-k5xbp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:03.684408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:03.684331    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84556a7c-211f-48e3-9211-ebc8bc5b79d5-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.203513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:04.203475    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"
Oct 30 19:40:04.203729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:04.203530    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9" event={"ID":"84556a7c-211f-48e3-9211-ebc8bc5b79d5","Type":"ContainerDied","Data":"82a55e1e02946a4ea53795221de3942f5a1657e8afcbc02603e75e1065f8e597"}
Oct 30 19:40:04.203729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:04.203630    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a55e1e02946a4ea53795221de3942f5a1657e8afcbc02603e75e1065f8e597"
Oct 30 19:40:04.622308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:04.622266    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:40:04.622767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:04.622333    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 19:40:05.622342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:05.622300    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:40:05.622884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:05.622370    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 19:40:06.628826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:06.628792    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:40:06.630784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:06.629312    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:40:06.630784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:06.630184    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:40:09.419443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:09.419351    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"]
Oct 30 19:40:09.419959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:09.419626    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy" containerID="cri-o://4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14" gracePeriod=30
Oct 30 19:40:12.278025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:12.277973    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp" podUID="87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:40:12.278025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:12.278015    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f" podUID="cb9152f3-269a-46bc-b268-7a8173084433" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:40:13.684453 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:40:13.684408    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:40:14.871185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.871148    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:40:14.911592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.911549    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-credential-socket\") pod \"08eb5014-7d21-4071-a25e-67d977c69bd0\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") "
Oct 30 19:40:14.912214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.911604    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-envoy\") pod \"08eb5014-7d21-4071-a25e-67d977c69bd0\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") "
Oct 30 19:40:14.912214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.911652    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-certs\") pod \"08eb5014-7d21-4071-a25e-67d977c69bd0\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") "
Oct 30 19:40:14.912214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.911678    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-socket\") pod \"08eb5014-7d21-4071-a25e-67d977c69bd0\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") "
Oct 30 19:40:14.912214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.911817    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-credential-socket" pod "08eb5014-7d21-4071-a25e-67d977c69bd0" (UID: "08eb5014-7d21-4071-a25e-67d977c69bd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:14.912214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.912004    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-token\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-token\") pod \"08eb5014-7d21-4071-a25e-67d977c69bd0\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") "
Oct 30 19:40:14.912214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.912143    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-kube-api-access-rfl25\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-kube-api-access-rfl25\") pod \"08eb5014-7d21-4071-a25e-67d977c69bd0\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") "
Oct 30 19:40:14.912571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.912275    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08eb5014-7d21-4071-a25e-67d977c69bd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb5014-7d21-4071-a25e-67d977c69bd0-istiod-ca-cert\") pod \"08eb5014-7d21-4071-a25e-67d977c69bd0\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") "
Oct 30 19:40:14.912571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.912324    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-podinfo\") pod \"08eb5014-7d21-4071-a25e-67d977c69bd0\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") "
Oct 30 19:40:14.912571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.912350    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-data\") pod \"08eb5014-7d21-4071-a25e-67d977c69bd0\" (UID: \"08eb5014-7d21-4071-a25e-67d977c69bd0\") "
Oct 30 19:40:14.913727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.913030    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"]
Oct 30 19:40:14.913727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.913236    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:14.913727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.913447    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-data" pod "08eb5014-7d21-4071-a25e-67d977c69bd0" (UID: "08eb5014-7d21-4071-a25e-67d977c69bd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:14.913727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.913602    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-certs" pod "08eb5014-7d21-4071-a25e-67d977c69bd0" (UID: "08eb5014-7d21-4071-a25e-67d977c69bd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:14.914424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.913749    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-socket" pod "08eb5014-7d21-4071-a25e-67d977c69bd0" (UID: "08eb5014-7d21-4071-a25e-67d977c69bd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:14.914424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.914188    6777 cpu_manager.go:414] "containerMap: removing container" podUID="84556a7c-211f-48e3-9211-ebc8bc5b79d5" containerName="revoke-keys"
Oct 30 19:40:14.914424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.914208    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84556a7c-211f-48e3-9211-ebc8bc5b79d5" containerName="revoke-keys"
Oct 30 19:40:14.914424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.914381    6777 cpu_manager.go:414] "containerMap: removing container" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy"
Oct 30 19:40:14.914424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.914394    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy"
Oct 30 19:40:14.914424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.914412    6777 cpu_manager.go:414] "containerMap: removing container" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy"
Oct 30 19:40:14.914424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.914421    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy"
Oct 30 19:40:14.915416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.914654    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy"
Oct 30 19:40:14.915416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.914675    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerName="istio-proxy"
Oct 30 19:40:14.915416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.914692    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="84556a7c-211f-48e3-9211-ebc8bc5b79d5" containerName="revoke-keys"
Oct 30 19:40:14.922961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.916462    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08eb5014-7d21-4071-a25e-67d977c69bd0-istiod-ca-cert" pod "08eb5014-7d21-4071-a25e-67d977c69bd0" (UID: "08eb5014-7d21-4071-a25e-67d977c69bd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:14.922961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.919034    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-kube-api-access-rfl25" pod "08eb5014-7d21-4071-a25e-67d977c69bd0" (UID: "08eb5014-7d21-4071-a25e-67d977c69bd0"). InnerVolumeSpecName "kube-api-access-rfl25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:14.922961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.919887    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-token" pod "08eb5014-7d21-4071-a25e-67d977c69bd0" (UID: "08eb5014-7d21-4071-a25e-67d977c69bd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:14.922961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.920050    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-envoy" pod "08eb5014-7d21-4071-a25e-67d977c69bd0" (UID: "08eb5014-7d21-4071-a25e-67d977c69bd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:14.922961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.921485    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:14.922961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.922901    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-podinfo" pod "08eb5014-7d21-4071-a25e-67d977c69bd0" (UID: "08eb5014-7d21-4071-a25e-67d977c69bd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:40:14.926861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.926825    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7e6199-serving-cert\""
Oct 30 19:40:14.927762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:14.927714    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"]
Oct 30 19:40:15.014960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.014916    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66rsr\" (UniqueName: \"kubernetes.io/projected/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-kube-api-access-66rsr\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.015197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015160    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.015350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015332    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.015424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015415    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.015667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015650    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfl25\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-kube-api-access-rfl25\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.015747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015672    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb5014-7d21-4071-a25e-67d977c69bd0-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.015747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015686    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.015747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015699    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.015747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015711    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.015747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015725    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.015747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015738    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb5014-7d21-4071-a25e-67d977c69bd0-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.016092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.015752    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb5014-7d21-4071-a25e-67d977c69bd0-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.117129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.117010    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.117355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.117246    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66rsr\" (UniqueName: \"kubernetes.io/projected/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-kube-api-access-66rsr\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.117437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.117387    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.117686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.117549    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.119190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.119158    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.121594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.121567    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.128658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.128621    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.130959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.130921    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66rsr\" (UniqueName: \"kubernetes.io/projected/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-kube-api-access-66rsr\") pod \"maas-api-e2e-ipp-a-7e6199-77447465d-9dncx\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.309635 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.309604    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:15.496851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.496806    6777 generic.go:356] "Generic (PLEG): container finished" podID="08eb5014-7d21-4071-a25e-67d977c69bd0" containerID="4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14" exitCode=0
Oct 30 19:40:15.497067 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.496874    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" event={"ID":"08eb5014-7d21-4071-a25e-67d977c69bd0","Type":"ContainerDied","Data":"4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14"}
Oct 30 19:40:15.497067 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.496910    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"
Oct 30 19:40:15.497067 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.496956    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq" event={"ID":"08eb5014-7d21-4071-a25e-67d977c69bd0","Type":"ContainerDied","Data":"7f262d4119b8d970d4df28c91ac1adaaa438018917a6827d9aa26369775c6fbb"}
Oct 30 19:40:15.497067 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.497002    6777 scope.go:122] "RemoveContainer" containerID="4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14"
Oct 30 19:40:15.515462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.514712    6777 scope.go:122] "RemoveContainer" containerID="c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b"
Oct 30 19:40:15.535831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.535738    6777 scope.go:122] "RemoveContainer" containerID="4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14"
Oct 30 19:40:15.541630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.541600    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"]
Oct 30 19:40:15.544556 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:40:15.544520    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14\": container with ID starting with 4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14 not found: ID does not exist" containerID="4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14"
Oct 30 19:40:15.544807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.544567    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14"} err="failed to get container status \"4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14\": rpc error: code = NotFound desc = could not find container \"4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14\": container with ID starting with 4edc78c5ee8d4e3cf0fcee9f25f21cae9a20a5370a4e500e2b5ef772d9b3ef14 not found: ID does not exist"
Oct 30 19:40:15.544807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.544596    6777 scope.go:122] "RemoveContainer" containerID="c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b"
Oct 30 19:40:15.546027 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:40:15.545383    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b\": container with ID starting with c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b not found: ID does not exist" containerID="c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b"
Oct 30 19:40:15.546027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.545420    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b"} err="failed to get container status \"c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b\": rpc error: code = NotFound desc = could not find container \"c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b\": container with ID starting with c338bac5608f3453d2ee4bcd68e4ae40f6c751010a10f0edf2a7469ba159e44b not found: ID does not exist"
Oct 30 19:40:15.547069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.547039    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-ndcjq"]
Oct 30 19:40:15.609968 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:40:15.609937    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3aff27e_b7d6_4e22_b0bb_80b8b25e26d4.slice/crio-1b6e435113159a6528480b8999e758a69d887db7f5627b992a47be9be7643dc7 WatchSource:0}: Error finding container 1b6e435113159a6528480b8999e758a69d887db7f5627b992a47be9be7643dc7: Status 404 returned error can't find the container with id 1b6e435113159a6528480b8999e758a69d887db7f5627b992a47be9be7643dc7
Oct 30 19:40:15.611034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.611005    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"]
Oct 30 19:40:15.612559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:15.612537    6777 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:40:16.118208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:16.118175    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08eb5014-7d21-4071-a25e-67d977c69bd0" path="/var/lib/kubelet/pods/08eb5014-7d21-4071-a25e-67d977c69bd0/volumes"
Oct 30 19:40:16.508865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:16.508759    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx" event={"ID":"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4","Type":"ContainerStarted","Data":"e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e"}
Oct 30 19:40:16.508865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:16.508830    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:16.508865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:16.508854    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx" event={"ID":"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4","Type":"ContainerStarted","Data":"1b6e435113159a6528480b8999e758a69d887db7f5627b992a47be9be7643dc7"}
Oct 30 19:40:16.536004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:16.535917    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx" podStartSLOduration=3.989136311 podStartE2EDuration="4.53589594s" totalImagesPullingTime="546.759629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:16.530244879 +0000 UTC m=+1998.527160266" watchObservedRunningTime="2026-10-30 19:40:16.53589594 +0000 UTC m=+1998.532811327"
Oct 30 19:40:19.523008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.522965    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:40:19.547483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.547437    6777 generic.go:356] "Generic (PLEG): container finished" podID="87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" containerID="5ebc2f7b4f64e0718dcc4b4a524702cee1f0cd0c438c535376376569032d348d" exitCode=0
Oct 30 19:40:19.547696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.547508    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp" event={"ID":"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb","Type":"ContainerDied","Data":"5ebc2f7b4f64e0718dcc4b4a524702cee1f0cd0c438c535376376569032d348d"}
Oct 30 19:40:19.547696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.547575    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp" event={"ID":"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb","Type":"ContainerDied","Data":"95d278e907d14cee1618592a72d79281f4831820d51b6d373b310764125e969f"}
Oct 30 19:40:19.547696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.547609    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95d278e907d14cee1618592a72d79281f4831820d51b6d373b310764125e969f"
Oct 30 19:40:19.549815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.549463    6777 generic.go:356] "Generic (PLEG): container finished" podID="cb9152f3-269a-46bc-b268-7a8173084433" containerID="43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21" exitCode=0
Oct 30 19:40:19.549815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.549514    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"
Oct 30 19:40:19.549815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.549686    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f" event={"ID":"cb9152f3-269a-46bc-b268-7a8173084433","Type":"ContainerDied","Data":"43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21"}
Oct 30 19:40:19.549815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.549724    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f" event={"ID":"cb9152f3-269a-46bc-b268-7a8173084433","Type":"ContainerDied","Data":"238cf0545631798729a65d935ef2eb0658983cffde3c308fc50046caa2be438b"}
Oct 30 19:40:19.549815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.549750    6777 scope.go:122] "RemoveContainer" containerID="43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21"
Oct 30 19:40:19.575908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.575879    6777 scope.go:122] "RemoveContainer" containerID="43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21"
Oct 30 19:40:19.576308 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:40:19.576277    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21\": container with ID starting with 43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21 not found: ID does not exist" containerID="43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21"
Oct 30 19:40:19.576478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.576317    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21"} err="failed to get container status \"43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21\": rpc error: code = NotFound desc = could not find container \"43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21\": container with ID starting with 43c8c7afb25f8e7e0d4c63abaf1b88b95b89419269ce267c55d5c76863039d21 not found: ID does not exist"
Oct 30 19:40:19.579050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.578979    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:40:19.600581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.598945    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9152f3-269a-46bc-b268-7a8173084433-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9152f3-269a-46bc-b268-7a8173084433-plugins-config-volume\") pod \"cb9152f3-269a-46bc-b268-7a8173084433\" (UID: \"cb9152f3-269a-46bc-b268-7a8173084433\") "
Oct 30 19:40:19.600581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.599046    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9152f3-269a-46bc-b268-7a8173084433-kube-api-access-w7vhf\" (UniqueName: \"kubernetes.io/projected/cb9152f3-269a-46bc-b268-7a8173084433-kube-api-access-w7vhf\") pod \"cb9152f3-269a-46bc-b268-7a8173084433\" (UID: \"cb9152f3-269a-46bc-b268-7a8173084433\") "
Oct 30 19:40:19.600581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.599612    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9152f3-269a-46bc-b268-7a8173084433-plugins-config-volume" pod "cb9152f3-269a-46bc-b268-7a8173084433" (UID: "cb9152f3-269a-46bc-b268-7a8173084433"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:19.600581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.600549    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9152f3-269a-46bc-b268-7a8173084433-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:19.603190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.603155    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9152f3-269a-46bc-b268-7a8173084433-kube-api-access-w7vhf" pod "cb9152f3-269a-46bc-b268-7a8173084433" (UID: "cb9152f3-269a-46bc-b268-7a8173084433"). InnerVolumeSpecName "kube-api-access-w7vhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:19.701809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.701768    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-kube-api-access-7frqh\" (UniqueName: \"kubernetes.io/projected/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-kube-api-access-7frqh\") pod \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\" (UID: \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\") "
Oct 30 19:40:19.702007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.701886    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-plugins-config-volume\") pod \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\" (UID: \"87bf8659-2bac-4aa4-8af3-3a0801fd1ccb\") "
Oct 30 19:40:19.702959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.702330    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-plugins-config-volume" pod "87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" (UID: "87bf8659-2bac-4aa4-8af3-3a0801fd1ccb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:19.702959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.702906    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7vhf\" (UniqueName: \"kubernetes.io/projected/cb9152f3-269a-46bc-b268-7a8173084433-kube-api-access-w7vhf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:19.702959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.702924    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:19.704254 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.704234    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-kube-api-access-7frqh" pod "87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" (UID: "87bf8659-2bac-4aa4-8af3-3a0801fd1ccb"). InnerVolumeSpecName "kube-api-access-7frqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:19.805297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.805260    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7frqh\" (UniqueName: \"kubernetes.io/projected/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb-kube-api-access-7frqh\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:19.881173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.881136    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"]
Oct 30 19:40:19.899381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:19.899338    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1fa500-6cb9b7dc56-2lh5f"]
Oct 30 19:40:20.115581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:20.115498    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9152f3-269a-46bc-b268-7a8173084433" path="/var/lib/kubelet/pods/cb9152f3-269a-46bc-b268-7a8173084433/volumes"
Oct 30 19:40:20.564484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:20.564448    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"
Oct 30 19:40:20.599187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:20.598533    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"]
Oct 30 19:40:20.603747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:20.603718    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1fa500-7ff4f97659-9mwbp"]
Oct 30 19:40:22.118455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:22.118421    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" path="/var/lib/kubelet/pods/87bf8659-2bac-4aa4-8af3-3a0801fd1ccb/volumes"
Oct 30 19:40:22.528328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:22.528252    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:40:32.253599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:32.253556    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:40:34.978274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:34.978209    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:34.980272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:34.980234    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.001888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.001842    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:40:35.192137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.186083    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:40:35.192137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.188204    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.192137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.188586    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": context deadline exceeded"
Oct 30 19:40:35.217081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.211185    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.257678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.234849    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.268101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.265157    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.559089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.549401    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.584081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.575884    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.584081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.576315    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:40:35.584081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.577308    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.070161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:35.994694    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.070161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.021210    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:40:36.070161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.049347    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.070161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.050238    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:40:36.223838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.194093    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.256457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.242710    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.256457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.243804    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.256457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.244143    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:40:36.307083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.292113    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.312815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.312213    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:40:36.335623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.335515    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:40:36.359580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.359542    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:40:36.360433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.360412    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:40:36.380176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.380048    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:40:36.603722 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.603598    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.810099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.781213    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:40:36.810099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:36.781921    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:40:37.152235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.152187    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.206:8000/ready\": context deadline exceeded"
Oct 30 19:40:37.162528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.162477    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:40:37.173121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.172201    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": context deadline exceeded"
Oct 30 19:40:37.173801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.173756    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:40:37.174647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.174029    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:40:37.174647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.174136    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:40:37.174647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.174312    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:40:37.174647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.174371    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:40:37.177793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.177763    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:40:37.212244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.212112    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:40:37.214653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.214625    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:40:37.216286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.216168    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:40:37.223336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.223294    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.318208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.303990    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:40:37.319612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.319570    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.373679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.373645    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:40:37.381342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.381069    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:40:37.382122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.382098    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:40:37.387493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.387467    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:40:37.387785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.387769    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:40:37.416306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:37.416215    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:40:39.044869 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.044830    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"]
Oct 30 19:40:39.046189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.046157    6777 cpu_manager.go:414] "containerMap: removing container" podUID="87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" containerName="payload-pre-processing"
Oct 30 19:40:39.046337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.046293    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" containerName="payload-pre-processing"
Oct 30 19:40:39.046337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.046332    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cb9152f3-269a-46bc-b268-7a8173084433" containerName="payload-processing"
Oct 30 19:40:39.046471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.046341    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9152f3-269a-46bc-b268-7a8173084433" containerName="payload-processing"
Oct 30 19:40:39.046604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.046586    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9152f3-269a-46bc-b268-7a8173084433" containerName="payload-processing"
Oct 30 19:40:39.046668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.046625    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="87bf8659-2bac-4aa4-8af3-3a0801fd1ccb" containerName="payload-pre-processing"
Oct 30 19:40:39.055885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.055849    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.060013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.059983    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"]
Oct 30 19:40:39.074404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.068409    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ddb6ea\"/\"kube-root-ca.crt\""
Oct 30 19:40:39.074404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.068416    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ddb6ea\"/\"rate-test-model-ddb6ea-kserve-self-signed-certs\""
Oct 30 19:40:39.074404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.068484    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ddb6ea\"/\"default-dockercfg-qlgfh\""
Oct 30 19:40:39.074404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.068593    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ddb6ea\"/\"openshift-service-ca.crt\""
Oct 30 19:40:39.192745 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.192701    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tls-certs\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.192979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.192894    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-home\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.192979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.192964    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-dshm\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.193154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.192993    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tmp-dir\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.193329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.193309    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-model-cache\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.193484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.193462    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5llv\" (UniqueName: \"kubernetes.io/projected/7dea1f43-5e0f-4c69-983a-e3c9f166d224-kube-api-access-f5llv\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.312141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.295426    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tls-certs\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.312141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.295528    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-home\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.312141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.295581    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-dshm\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.312141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.295606    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tmp-dir\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.312141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.295741    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-model-cache\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.312141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.295865    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5llv\" (UniqueName: \"kubernetes.io/projected/7dea1f43-5e0f-4c69-983a-e3c9f166d224-kube-api-access-f5llv\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.330229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.318583    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-home\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.330229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.321153    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-model-cache\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.330229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.321890    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tmp-dir\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.330229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.326549    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-dshm\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.330229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.327825    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5llv\" (UniqueName: \"kubernetes.io/projected/7dea1f43-5e0f-4c69-983a-e3c9f166d224-kube-api-access-f5llv\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.384172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.377978    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tls-certs\") pod \"rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.384172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.381605    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:39.800175 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:40:39.800142    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dea1f43_5e0f_4c69_983a_e3c9f166d224.slice/crio-f2823e9f2f4886551ef3e924e7a2ae86221987c32cc9c565631f0f230ab2d139 WatchSource:0}: Error finding container f2823e9f2f4886551ef3e924e7a2ae86221987c32cc9c565631f0f230ab2d139: Status 404 returned error can't find the container with id f2823e9f2f4886551ef3e924e7a2ae86221987c32cc9c565631f0f230ab2d139
Oct 30 19:40:39.801312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:39.801281    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"]
Oct 30 19:40:40.454460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:40.454411    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" event={"ID":"7dea1f43-5e0f-4c69-983a-e3c9f166d224","Type":"ContainerStarted","Data":"9f56274d07464ed90c85a93c33e4196a14c932c53005f4c6488fcd6c33f4c5b4"}
Oct 30 19:40:40.455025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:40.454471    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" event={"ID":"7dea1f43-5e0f-4c69-983a-e3c9f166d224","Type":"ContainerStarted","Data":"f2823e9f2f4886551ef3e924e7a2ae86221987c32cc9c565631f0f230ab2d139"}
Oct 30 19:40:40.477823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:40.477763    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" podStartSLOduration=1.477749373 podStartE2EDuration="1.477749373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:40.476396304 +0000 UTC m=+2022.473311697" watchObservedRunningTime="2026-10-30 19:40:40.477749373 +0000 UTC m=+2022.474664758"
Oct 30 19:40:41.386081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:41.385791    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:43.643615 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:40:43.643569    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:40:45.023667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:45.023632    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"]
Oct 30 19:40:49.387079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:49.386038    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:49.483791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:49.483756    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:49.564376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:49.564338    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:40:50.415511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:50.415474    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"]
Oct 30 19:40:52.925020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:52.924984    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"]
Oct 30 19:40:52.929943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:52.929918    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:40:52.940398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:52.940364    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"]
Oct 30 19:40:52.942287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:52.942252    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e6199\""
Oct 30 19:40:52.943638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:52.943426    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7e6199-dockercfg-9kpn4\""
Oct 30 19:40:53.237149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:53.209510    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc642a52-d707-409a-9529-a0e843a06c66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss\" (UID: \"fc642a52-d707-409a-9529-a0e843a06c66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:40:53.237149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:53.231848    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdjn2\" (UniqueName: \"kubernetes.io/projected/fc642a52-d707-409a-9529-a0e843a06c66-kube-api-access-sdjn2\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss\" (UID: \"fc642a52-d707-409a-9529-a0e843a06c66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:40:53.342079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:53.341048    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc642a52-d707-409a-9529-a0e843a06c66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss\" (UID: \"fc642a52-d707-409a-9529-a0e843a06c66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:40:53.342079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:53.341378    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdjn2\" (UniqueName: \"kubernetes.io/projected/fc642a52-d707-409a-9529-a0e843a06c66-kube-api-access-sdjn2\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss\" (UID: \"fc642a52-d707-409a-9529-a0e843a06c66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:40:53.342680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:53.342613    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc642a52-d707-409a-9529-a0e843a06c66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss\" (UID: \"fc642a52-d707-409a-9529-a0e843a06c66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:40:53.355807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:53.355764    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdjn2\" (UniqueName: \"kubernetes.io/projected/fc642a52-d707-409a-9529-a0e843a06c66-kube-api-access-sdjn2\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss\" (UID: \"fc642a52-d707-409a-9529-a0e843a06c66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:40:53.359006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:53.358957    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:40:53.675659 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:40:53.675625    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc642a52_d707_409a_9529_a0e843a06c66.slice/crio-ec41b55ceea66702a20064cb4048e5b814e15ed3be912eb4ed5db6dc145def3d WatchSource:0}: Error finding container ec41b55ceea66702a20064cb4048e5b814e15ed3be912eb4ed5db6dc145def3d: Status 404 returned error can't find the container with id ec41b55ceea66702a20064cb4048e5b814e15ed3be912eb4ed5db6dc145def3d
Oct 30 19:40:53.676117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:53.676092    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"]
Oct 30 19:40:54.639239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:54.639199    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss" event={"ID":"fc642a52-d707-409a-9529-a0e843a06c66","Type":"ContainerStarted","Data":"ec41b55ceea66702a20064cb4048e5b814e15ed3be912eb4ed5db6dc145def3d"}
Oct 30 19:40:55.650813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:55.650775    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss" event={"ID":"fc642a52-d707-409a-9529-a0e843a06c66","Type":"ContainerStarted","Data":"d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5"}
Oct 30 19:40:55.651424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:55.650839    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:40:55.727720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:40:55.727568    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss" podStartSLOduration=1.9369627889999999 podStartE2EDuration="3.727550109s" totalImagesPullingTime="1.79058732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:55.726331451 +0000 UTC m=+2037.723246837" watchObservedRunningTime="2026-10-30 19:40:55.727550109 +0000 UTC m=+2037.724465534"
Oct 30 19:41:00.065052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.065011    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" containerID="cri-o://f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326" gracePeriod=45
Oct 30 19:41:00.087393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.087357    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": dial tcp 10.134.0.199:8000: connect: connection refused"
Oct 30 19:41:00.379921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.379875    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:41:00.537076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.537026    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-model-cache\") pod \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") "
Oct 30 19:41:00.537271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.537130    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tls-certs\") pod \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") "
Oct 30 19:41:00.537271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.537207    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-dshm\") pod \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") "
Oct 30 19:41:00.537347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.537320    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-home\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-home\") pod \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") "
Oct 30 19:41:00.537381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.537362    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tmp-dir\") pod \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") "
Oct 30 19:41:00.537432 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.537373    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-model-cache" pod "c7669ea1-4f99-4316-be8c-54ce10d66e0a" (UID: "c7669ea1-4f99-4316-be8c-54ce10d66e0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:00.538071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.537597    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-home" pod "c7669ea1-4f99-4316-be8c-54ce10d66e0a" (UID: "c7669ea1-4f99-4316-be8c-54ce10d66e0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:00.538071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.537653    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tmp-dir" pod "c7669ea1-4f99-4316-be8c-54ce10d66e0a" (UID: "c7669ea1-4f99-4316-be8c-54ce10d66e0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:00.538071 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.537656    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7669ea1-4f99-4316-be8c-54ce10d66e0a-kube-api-access-57t5k\" (UniqueName: \"kubernetes.io/projected/c7669ea1-4f99-4316-be8c-54ce10d66e0a-kube-api-access-57t5k\") pod \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\" (UID: \"c7669ea1-4f99-4316-be8c-54ce10d66e0a\") "
Oct 30 19:41:00.538511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.538490    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:00.538511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.538511    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:00.538655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.538519    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:00.539600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.539568    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tls-certs" pod "c7669ea1-4f99-4316-be8c-54ce10d66e0a" (UID: "c7669ea1-4f99-4316-be8c-54ce10d66e0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:00.539697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.539641    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7669ea1-4f99-4316-be8c-54ce10d66e0a-kube-api-access-57t5k" pod "c7669ea1-4f99-4316-be8c-54ce10d66e0a" (UID: "c7669ea1-4f99-4316-be8c-54ce10d66e0a"). InnerVolumeSpecName "kube-api-access-57t5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:00.539697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.539674    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-dshm" pod "c7669ea1-4f99-4316-be8c-54ce10d66e0a" (UID: "c7669ea1-4f99-4316-be8c-54ce10d66e0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:00.640279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.640191    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57t5k\" (UniqueName: \"kubernetes.io/projected/c7669ea1-4f99-4316-be8c-54ce10d66e0a-kube-api-access-57t5k\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:00.640279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.640224    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7669ea1-4f99-4316-be8c-54ce10d66e0a-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:00.640279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.640234    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7669ea1-4f99-4316-be8c-54ce10d66e0a-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:00.700973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.700940    6777 generic.go:356] "Generic (PLEG): container finished" podID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerID="f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326" exitCode=0
Oct 30 19:41:00.701243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.700991    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"
Oct 30 19:41:00.701243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.701015    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" event={"ID":"c7669ea1-4f99-4316-be8c-54ce10d66e0a","Type":"ContainerDied","Data":"f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326"}
Oct 30 19:41:00.701243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.701077    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85" event={"ID":"c7669ea1-4f99-4316-be8c-54ce10d66e0a","Type":"ContainerDied","Data":"1f8df2dab9898a7d10ce01a9e14ce24ed7b61d7f92c635d715bac7c45379bc36"}
Oct 30 19:41:00.701243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.701103    6777 scope.go:122] "RemoveContainer" containerID="f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326"
Oct 30 19:41:00.720953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.720929    6777 scope.go:122] "RemoveContainer" containerID="f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326"
Oct 30 19:41:00.721361 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:41:00.721338    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326\": container with ID starting with f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326 not found: ID does not exist" containerID="f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326"
Oct 30 19:41:00.721403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.721374    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326"} err="failed to get container status \"f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326\": rpc error: code = NotFound desc = could not find container \"f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326\": container with ID starting with f624fb4f519d526555a63e1b9bcfb3bd0edff9c2ef129b1ec64b0c595f391326 not found: ID does not exist"
Oct 30 19:41:00.732912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.732879    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"]
Oct 30 19:41:00.737690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:00.737655    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2405fb/sub-test-model-2405fb-kserve-684c4577c9-z8q85"]
Oct 30 19:41:01.246391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:01.246355    6777 scope.go:122] "RemoveContainer" containerID="3e4c9f4400fe1735ebceb2367360ec1fa51df8d022359d8da8ba10a9c601f80a"
Oct 30 19:41:01.323873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:01.323818    6777 scope.go:122] "RemoveContainer" containerID="40cf458f41665c377b90c9dbdd7496cb9d894d85fe2075d826d7da8b705ff02b"
Oct 30 19:41:01.340302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:01.340277    6777 scope.go:122] "RemoveContainer" containerID="c01821dfbefea79f4ed2c39d29a9a181041c23009305cbb4045768dcee8408b6"
Oct 30 19:41:01.354845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:01.354822    6777 scope.go:122] "RemoveContainer" containerID="df804c565fd63b9045b6190b3c4ed04bcf6090516c5c11886793da2cfa4cd4e3"
Oct 30 19:41:01.368216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:01.368195    6777 scope.go:122] "RemoveContainer" containerID="5ebc2f7b4f64e0718dcc4b4a524702cee1f0cd0c438c535376376569032d348d"
Oct 30 19:41:01.382467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:01.381827    6777 scope.go:122] "RemoveContainer" containerID="64c195d9927e1ca00bbbcb10e653a29c9f9c05ef3dbb797a06bc4198d3080388"
Oct 30 19:41:01.400370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:01.400331    6777 scope.go:122] "RemoveContainer" containerID="e15f2ad651a81e92256339292512b5f732e3c735eaee6b46fafaf8b059929865"
Oct 30 19:41:01.413356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:01.412679    6777 scope.go:122] "RemoveContainer" containerID="1f9897283ed010cf16f7cae3623ce7e88b4bcd237f9bbb5a926da91ca285eeff"
Oct 30 19:41:02.117794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:02.117754    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" path="/var/lib/kubelet/pods/c7669ea1-4f99-4316-be8c-54ce10d66e0a/volumes"
Oct 30 19:41:05.446968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.446916    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main" containerID="cri-o://f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7" gracePeriod=45
Oct 30 19:41:05.780769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.780734    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:41:05.783663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.783631    6777 generic.go:356] "Generic (PLEG): container finished" podID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerID="f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7" exitCode=0
Oct 30 19:41:05.783663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.783660    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"
Oct 30 19:41:05.783879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.783696    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" event={"ID":"bfae7bd8-8b51-4867-ac02-33beb7c228f9","Type":"ContainerDied","Data":"f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7"}
Oct 30 19:41:05.783879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.783739    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks" event={"ID":"bfae7bd8-8b51-4867-ac02-33beb7c228f9","Type":"ContainerDied","Data":"42294f7f81d4a0249ca849048e3b8fb667085b1853791b71af012a5a42fda4d8"}
Oct 30 19:41:05.783879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.783756    6777 scope.go:122] "RemoveContainer" containerID="f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7"
Oct 30 19:41:05.840327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.840281    6777 scope.go:122] "RemoveContainer" containerID="f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7"
Oct 30 19:41:05.848106 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:41:05.847276    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7\": container with ID starting with f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7 not found: ID does not exist" containerID="f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7"
Oct 30 19:41:05.848106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.847332    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7"} err="failed to get container status \"f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7\": rpc error: code = NotFound desc = could not find container \"f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7\": container with ID starting with f1fde278e8260f95dc1f9fce63d5768ea12dabd28058c0533db14144beb39cb7 not found: ID does not exist"
Oct 30 19:41:05.862135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.862099    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-dshm\") pod \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") "
Oct 30 19:41:05.862480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.862421    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfae7bd8-8b51-4867-ac02-33beb7c228f9-kube-api-access-stk4d\" (UniqueName: \"kubernetes.io/projected/bfae7bd8-8b51-4867-ac02-33beb7c228f9-kube-api-access-stk4d\") pod \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") "
Oct 30 19:41:05.863101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.862559    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-model-cache\") pod \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") "
Oct 30 19:41:05.863101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.862604    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-home\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-home\") pod \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") "
Oct 30 19:41:05.863101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.862639    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tls-certs\") pod \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") "
Oct 30 19:41:05.863101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.862774    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tmp-dir\") pod \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\" (UID: \"bfae7bd8-8b51-4867-ac02-33beb7c228f9\") "
Oct 30 19:41:05.864349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.864308    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-model-cache" pod "bfae7bd8-8b51-4867-ac02-33beb7c228f9" (UID: "bfae7bd8-8b51-4867-ac02-33beb7c228f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:05.865243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.864517    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-home" pod "bfae7bd8-8b51-4867-ac02-33beb7c228f9" (UID: "bfae7bd8-8b51-4867-ac02-33beb7c228f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:05.866194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.866152    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tmp-dir" pod "bfae7bd8-8b51-4867-ac02-33beb7c228f9" (UID: "bfae7bd8-8b51-4867-ac02-33beb7c228f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:05.873266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.870461    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tls-certs" pod "bfae7bd8-8b51-4867-ac02-33beb7c228f9" (UID: "bfae7bd8-8b51-4867-ac02-33beb7c228f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:05.873266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.871902    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-dshm" pod "bfae7bd8-8b51-4867-ac02-33beb7c228f9" (UID: "bfae7bd8-8b51-4867-ac02-33beb7c228f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:05.880531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.878691    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfae7bd8-8b51-4867-ac02-33beb7c228f9-kube-api-access-stk4d" pod "bfae7bd8-8b51-4867-ac02-33beb7c228f9" (UID: "bfae7bd8-8b51-4867-ac02-33beb7c228f9"). InnerVolumeSpecName "kube-api-access-stk4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:05.966203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.966151    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stk4d\" (UniqueName: \"kubernetes.io/projected/bfae7bd8-8b51-4867-ac02-33beb7c228f9-kube-api-access-stk4d\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:05.966203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.966201    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:05.966203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.966215    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:05.966514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.966229    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:05.966514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.966242    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:05.966514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:05.966253    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfae7bd8-8b51-4867-ac02-33beb7c228f9-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.118979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:06.118946    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"]
Oct 30 19:41:06.119187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:06.118994    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e7123/sub-test-model-1e7123-kserve-59c74fc4cf-srbks"]
Oct 30 19:41:06.665019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:06.664982    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:41:07.182639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.182589    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"]
Oct 30 19:41:07.183716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.183686    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main"
Oct 30 19:41:07.183716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.183714    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main"
Oct 30 19:41:07.183903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.183753    6777 cpu_manager.go:414] "containerMap: removing container" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main"
Oct 30 19:41:07.183903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.183759    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main"
Oct 30 19:41:07.183903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.183900    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" containerName="main"
Oct 30 19:41:07.184083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.183921    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7669ea1-4f99-4316-be8c-54ce10d66e0a" containerName="main"
Oct 30 19:41:07.190031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.190001    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.203244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.203213    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"]
Oct 30 19:41:07.205705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.205031    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd7c77\"/\"rate-test-model-fd7c77-kserve-self-signed-certs\""
Oct 30 19:41:07.205705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.205097    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd7c77\"/\"kube-root-ca.crt\""
Oct 30 19:41:07.206602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.206578    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd7c77\"/\"openshift-service-ca.crt\""
Oct 30 19:41:07.218122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.218088    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd7c77\"/\"default-dockercfg-d4txb\""
Oct 30 19:41:07.293506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.293460    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-model-cache\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.293763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.293545    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-tmp-dir\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.293763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.293587    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dcsl\" (UniqueName: \"kubernetes.io/projected/ef751334-a08e-4bd0-afd8-d700e073bfff-kube-api-access-7dcsl\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.293763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.293638    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef751334-a08e-4bd0-afd8-d700e073bfff-tls-certs\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.293763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.293716    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-dshm\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.293993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.293978    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-home\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.469139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.468258    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-home\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.469139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.468421    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-model-cache\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.469139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.468475    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-tmp-dir\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.469139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.468524    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dcsl\" (UniqueName: \"kubernetes.io/projected/ef751334-a08e-4bd0-afd8-d700e073bfff-kube-api-access-7dcsl\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.469139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.468579    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef751334-a08e-4bd0-afd8-d700e073bfff-tls-certs\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.469139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.468666    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-dshm\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.470365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.470298    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-home\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.481218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.481190    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-model-cache\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.481644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.481625    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-tmp-dir\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.542189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.533969    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef751334-a08e-4bd0-afd8-d700e073bfff-tls-certs\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.557415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.557367    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-dshm\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.595963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.595913    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dcsl\" (UniqueName: \"kubernetes.io/projected/ef751334-a08e-4bd0-afd8-d700e073bfff-kube-api-access-7dcsl\") pod \"rate-test-model-fd7c77-kserve-775d7d46d5-njbcw\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:07.813019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:07.812975    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:08.117328 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:41:08.117288    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef751334_a08e_4bd0_afd8_d700e073bfff.slice/crio-85f56d78e6bc1d6199711d2e4f9261c6b3f53286e5fa5359507a3b7fd27ad1e1 WatchSource:0}: Error finding container 85f56d78e6bc1d6199711d2e4f9261c6b3f53286e5fa5359507a3b7fd27ad1e1: Status 404 returned error can't find the container with id 85f56d78e6bc1d6199711d2e4f9261c6b3f53286e5fa5359507a3b7fd27ad1e1
Oct 30 19:41:08.117953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:08.117928    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfae7bd8-8b51-4867-ac02-33beb7c228f9" path="/var/lib/kubelet/pods/bfae7bd8-8b51-4867-ac02-33beb7c228f9/volumes"
Oct 30 19:41:08.119257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:08.119236    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"]
Oct 30 19:41:08.814040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:08.814003    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" event={"ID":"ef751334-a08e-4bd0-afd8-d700e073bfff","Type":"ContainerStarted","Data":"ca250a79893739698f92dc3331c32438020b4b16a2e546ef3bd6c054cf177d60"}
Oct 30 19:41:08.814588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:08.814052    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" event={"ID":"ef751334-a08e-4bd0-afd8-d700e073bfff","Type":"ContainerStarted","Data":"85f56d78e6bc1d6199711d2e4f9261c6b3f53286e5fa5359507a3b7fd27ad1e1"}
Oct 30 19:41:08.837806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:08.837743    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" podStartSLOduration=1.837723344 podStartE2EDuration="1.837723344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:08.83487191 +0000 UTC m=+2050.831787296" watchObservedRunningTime="2026-10-30 19:41:08.837723344 +0000 UTC m=+2050.834638729"
Oct 30 19:41:09.135581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:09.135477    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"]
Oct 30 19:41:09.813610 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:09.813569    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:10.308041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:10.308000    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"]
Oct 30 19:41:10.308622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:10.308254    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy" containerID="cri-o://c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572" gracePeriod=30
Oct 30 19:41:13.667624 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:41:13.667584    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:41:15.810557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.810527    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:41:15.850747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.850713    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"]
Oct 30 19:41:15.851632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.851615    6777 cpu_manager.go:414] "containerMap: removing container" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy"
Oct 30 19:41:15.851696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.851633    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy"
Oct 30 19:41:15.851819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.851810    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerName="istio-proxy"
Oct 30 19:41:15.856373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.856347    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:15.872831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.871304    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f5613a-openshift-default-dockercfg-t8fg8\""
Oct 30 19:41:15.877665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.877633    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"]
Oct 30 19:41:15.910453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.910407    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-data\") pod \"014ae06d-ddaf-41b9-a59a-543047e656fd\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") "
Oct 30 19:41:15.910645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.910566    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-credential-socket\") pod \"014ae06d-ddaf-41b9-a59a-543047e656fd\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") "
Oct 30 19:41:15.910716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.910700    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-envoy\") pod \"014ae06d-ddaf-41b9-a59a-543047e656fd\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") "
Oct 30 19:41:15.910808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.910792    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-kube-api-access-tdjln\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-kube-api-access-tdjln\") pod \"014ae06d-ddaf-41b9-a59a-543047e656fd\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") "
Oct 30 19:41:15.910858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.910846    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-token\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-token\") pod \"014ae06d-ddaf-41b9-a59a-543047e656fd\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") "
Oct 30 19:41:15.910918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.910904    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-podinfo\") pod \"014ae06d-ddaf-41b9-a59a-543047e656fd\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") "
Oct 30 19:41:15.911013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.910990    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-socket\") pod \"014ae06d-ddaf-41b9-a59a-543047e656fd\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") "
Oct 30 19:41:15.911126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.911042    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-certs\") pod \"014ae06d-ddaf-41b9-a59a-543047e656fd\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") "
Oct 30 19:41:15.911201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.911182    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014ae06d-ddaf-41b9-a59a-543047e656fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014ae06d-ddaf-41b9-a59a-543047e656fd-istiod-ca-cert\") pod \"014ae06d-ddaf-41b9-a59a-543047e656fd\" (UID: \"014ae06d-ddaf-41b9-a59a-543047e656fd\") "
Oct 30 19:41:15.912740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.912682    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-credential-socket" pod "014ae06d-ddaf-41b9-a59a-543047e656fd" (UID: "014ae06d-ddaf-41b9-a59a-543047e656fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:15.912740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.912665    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-socket" pod "014ae06d-ddaf-41b9-a59a-543047e656fd" (UID: "014ae06d-ddaf-41b9-a59a-543047e656fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:15.912946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.912841    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014ae06d-ddaf-41b9-a59a-543047e656fd-istiod-ca-cert" pod "014ae06d-ddaf-41b9-a59a-543047e656fd" (UID: "014ae06d-ddaf-41b9-a59a-543047e656fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:15.933037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.932992    6777 generic.go:356] "Generic (PLEG): container finished" podID="014ae06d-ddaf-41b9-a59a-543047e656fd" containerID="c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572" exitCode=0
Oct 30 19:41:15.933037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.933132    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" event={"ID":"014ae06d-ddaf-41b9-a59a-543047e656fd","Type":"ContainerDied","Data":"c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572"}
Oct 30 19:41:15.937376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.933193    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh" event={"ID":"014ae06d-ddaf-41b9-a59a-543047e656fd","Type":"ContainerDied","Data":"abc4b047e3624b7772f06d6ea07124ae414a99ceb17f07eef701bf317b61726c"}
Oct 30 19:41:15.937376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.933229    6777 scope.go:122] "RemoveContainer" containerID="c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572"
Oct 30 19:41:15.937376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.933918    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"
Oct 30 19:41:15.952658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.952608    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-data" pod "014ae06d-ddaf-41b9-a59a-543047e656fd" (UID: "014ae06d-ddaf-41b9-a59a-543047e656fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:15.953609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.953567    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-podinfo" pod "014ae06d-ddaf-41b9-a59a-543047e656fd" (UID: "014ae06d-ddaf-41b9-a59a-543047e656fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:41:15.953772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.953602    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-kube-api-access-tdjln" pod "014ae06d-ddaf-41b9-a59a-543047e656fd" (UID: "014ae06d-ddaf-41b9-a59a-543047e656fd"). InnerVolumeSpecName "kube-api-access-tdjln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:15.953772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.953584    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-envoy" pod "014ae06d-ddaf-41b9-a59a-543047e656fd" (UID: "014ae06d-ddaf-41b9-a59a-543047e656fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:15.954122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.954012    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-token" pod "014ae06d-ddaf-41b9-a59a-543047e656fd" (UID: "014ae06d-ddaf-41b9-a59a-543047e656fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:15.954274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:15.954186    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-certs" pod "014ae06d-ddaf-41b9-a59a-543047e656fd" (UID: "014ae06d-ddaf-41b9-a59a-543047e656fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.039680    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qn9f\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-kube-api-access-8qn9f\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.039788    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-socket\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.039821    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-data\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.039839    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-podinfo\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.039866    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-credential-socket\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.039957    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-certs\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.039972    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e937cd6-3dcd-4309-9494-3df69a56d551-istiod-ca-cert\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040000    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-envoy\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040016    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-token\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040177    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040194    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040209    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040221    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdjln\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-kube-api-access-tdjln\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040234    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040247    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014ae06d-ddaf-41b9-a59a-543047e656fd-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040259    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040271    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014ae06d-ddaf-41b9-a59a-543047e656fd-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:16.047136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.040283    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014ae06d-ddaf-41b9-a59a-543047e656fd-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:16.059502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.054791    6777 scope.go:122] "RemoveContainer" containerID="c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572"
Oct 30 19:41:16.059502 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:41:16.057039    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572\": container with ID starting with c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572 not found: ID does not exist" containerID="c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572"
Oct 30 19:41:16.059502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.057141    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572"} err="failed to get container status \"c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572\": rpc error: code = NotFound desc = could not find container \"c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572\": container with ID starting with c8f3c382063827e7e5a86340ebd88cc15e072002741f77d5ab33a715c9ba3572 not found: ID does not exist"
Oct 30 19:41:16.141981 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.141945    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-socket\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.141995    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-data\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142097    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-podinfo\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142134    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-credential-socket\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142230    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-certs\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142257    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e937cd6-3dcd-4309-9494-3df69a56d551-istiod-ca-cert\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142302    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-envoy\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142330    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-token\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142414    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-socket\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142440    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-data\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142508    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-credential-socket\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142638    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-certs\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.142809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.142743    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qn9f\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-kube-api-access-8qn9f\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.143641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.143336    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e937cd6-3dcd-4309-9494-3df69a56d551-istiod-ca-cert\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.145036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.144926    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-envoy\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.145185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.145113    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-podinfo\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.156787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.156749    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-token\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.156787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.156785    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qn9f\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-kube-api-access-8qn9f\") pod \"e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") " pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.178079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.178036    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:16.261123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.261079    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"]
Oct 30 19:41:16.269183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.269139    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-chjvh"]
Oct 30 19:41:16.484702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.483923    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"]
Oct 30 19:41:16.484702 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:41:16.484007    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e937cd6_3dcd_4309_9494_3df69a56d551.slice/crio-bee3e96be5de4b7c4516baddf067d644e67e674132703b6b6394d16622db7afb WatchSource:0}: Error finding container bee3e96be5de4b7c4516baddf067d644e67e674132703b6b6394d16622db7afb: Status 404 returned error can't find the container with id bee3e96be5de4b7c4516baddf067d644e67e674132703b6b6394d16622db7afb
Oct 30 19:41:16.486888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.486852    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:41:16.486979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.486927    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:41:16.486979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:16.486961    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:41:17.206011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:17.205962    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" event={"ID":"8e937cd6-3dcd-4309-9494-3df69a56d551","Type":"ContainerStarted","Data":"0e3232805a6b390d0e668734d511446ffb2394c76d5dc6bb676bc99d3fdbc4d0"}
Oct 30 19:41:17.207070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:17.206044    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" event={"ID":"8e937cd6-3dcd-4309-9494-3df69a56d551","Type":"ContainerStarted","Data":"bee3e96be5de4b7c4516baddf067d644e67e674132703b6b6394d16622db7afb"}
Oct 30 19:41:17.239543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:17.239481    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" podStartSLOduration=26.239464711 podStartE2EDuration="26.239464711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:17.235718452 +0000 UTC m=+2059.232633838" watchObservedRunningTime="2026-10-30 19:41:17.239464711 +0000 UTC m=+2059.236380096"
Oct 30 19:41:17.731713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:17.731663    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:41:17.822143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:17.820219    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:18.098510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:18.097353    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:18.127300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:18.127265    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014ae06d-ddaf-41b9-a59a-543047e656fd" path="/var/lib/kubelet/pods/014ae06d-ddaf-41b9-a59a-543047e656fd/volumes"
Oct 30 19:41:18.179175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:18.179137    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:18.185990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:18.185959    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:18.223108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:18.221549    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:18.223108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:18.221906    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:41:18.236875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:18.236836    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:41:20.867083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:20.866978    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:41:20.906219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:20.906178    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:41:20.906535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:20.906502    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:20.988230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:20.988178    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.215621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.215512    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.217872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.217831    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.409280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.218757    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.411252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.237198    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.236:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.411252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.237254    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:41:21.411252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.399437    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:41:21.411252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.399486    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.411252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.399507    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:41:21.603101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.574320    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.234:8000/ready\": context deadline exceeded"
Oct 30 19:41:21.730939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.707670    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": context deadline exceeded"
Oct 30 19:41:21.734043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.733991    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 30 19:41:21.923780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.923684    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"]
Oct 30 19:41:21.965646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.965611    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:21.997575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.980997    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f5613a\""
Oct 30 19:41:21.997575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:21.994373    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f5613a-dockercfg-wkrp9\""
Oct 30 19:41:22.007279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.006832    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.009729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.009626    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:41:22.054257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.054212    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc\" (UID: \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:22.055018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.054974    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhlc\" (UniqueName: \"kubernetes.io/projected/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-kube-api-access-fjhlc\") pod \"payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc\" (UID: \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:22.102204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.102160    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"]
Oct 30 19:41:22.263497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.263391    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhlc\" (UniqueName: \"kubernetes.io/projected/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-kube-api-access-fjhlc\") pod \"payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc\" (UID: \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:22.290142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.290026    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc\" (UID: \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:22.292508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.267540    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:41:22.300673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.300639    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:41:22.335741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.335706    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:41:22.336018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.335999    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"]
Oct 30 19:41:22.337815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.337188    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy" containerID="cri-o://3f192d20da9794a057741959b0f5ad644e7f2978395fb42912837efaff0590ab" gracePeriod=30
Oct 30 19:41:22.338620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.331031    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhlc\" (UniqueName: \"kubernetes.io/projected/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-kube-api-access-fjhlc\") pod \"payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc\" (UID: \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:22.339558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.325715    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc\" (UID: \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:22.364825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.364791    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:41:22.430320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.428406    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:22.843603 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:41:22.843564    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d9f55b_bf47_40ca_8715_0fd1cbbbb893.slice/crio-2f9044986715e23c6ac5c94f5c306447baeb4519b23dabf73d36dcec537d9829 WatchSource:0}: Error finding container 2f9044986715e23c6ac5c94f5c306447baeb4519b23dabf73d36dcec537d9829: Status 404 returned error can't find the container with id 2f9044986715e23c6ac5c94f5c306447baeb4519b23dabf73d36dcec537d9829
Oct 30 19:41:22.845483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:22.845031    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"]
Oct 30 19:41:23.526773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:23.526728    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc" event={"ID":"78d9f55b-bf47-40ca-8715-0fd1cbbbb893","Type":"ContainerStarted","Data":"895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495"}
Oct 30 19:41:23.527338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:23.526812    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:23.527338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:23.526837    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc" event={"ID":"78d9f55b-bf47-40ca-8715-0fd1cbbbb893","Type":"ContainerStarted","Data":"2f9044986715e23c6ac5c94f5c306447baeb4519b23dabf73d36dcec537d9829"}
Oct 30 19:41:23.552246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:23.552169    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc" podStartSLOduration=2.552147047 podStartE2EDuration="2.552147047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:23.5473354 +0000 UTC m=+2065.544250786" watchObservedRunningTime="2026-10-30 19:41:23.552147047 +0000 UTC m=+2065.549062433"
Oct 30 19:41:24.160771 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.160726    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" containerID="cri-o://a6f8c7c6783cba1587526b865245c4530665608b5cb9b4b4decf06c166bfb27a" gracePeriod=45
Oct 30 19:41:24.543762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.543726    6777 generic.go:356] "Generic (PLEG): container finished" podID="55866653-01c9-4307-9d74-cb117399e0d5" containerID="a6f8c7c6783cba1587526b865245c4530665608b5cb9b4b4decf06c166bfb27a" exitCode=0
Oct 30 19:41:24.544173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.543768    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" event={"ID":"55866653-01c9-4307-9d74-cb117399e0d5","Type":"ContainerDied","Data":"a6f8c7c6783cba1587526b865245c4530665608b5cb9b4b4decf06c166bfb27a"}
Oct 30 19:41:24.544173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.543810    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" event={"ID":"55866653-01c9-4307-9d74-cb117399e0d5","Type":"ContainerDied","Data":"6ee1ba425c24a8f623140d1a86f2698a0f92407a8cefc01c9fd8f3bbee893ec5"}
Oct 30 19:41:24.544173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.543828    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ee1ba425c24a8f623140d1a86f2698a0f92407a8cefc01c9fd8f3bbee893ec5"
Oct 30 19:41:24.570552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.570520    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:41:24.644009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.643964    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-model-cache\") pod \"55866653-01c9-4307-9d74-cb117399e0d5\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") "
Oct 30 19:41:24.644295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.644268    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-dshm\") pod \"55866653-01c9-4307-9d74-cb117399e0d5\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") "
Oct 30 19:41:24.644487 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.644470    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55866653-01c9-4307-9d74-cb117399e0d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/55866653-01c9-4307-9d74-cb117399e0d5-tls-certs\") pod \"55866653-01c9-4307-9d74-cb117399e0d5\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") "
Oct 30 19:41:24.644627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.644606    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-tmp-dir\") pod \"55866653-01c9-4307-9d74-cb117399e0d5\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") "
Oct 30 19:41:24.645182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.644654    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-home\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-home\") pod \"55866653-01c9-4307-9d74-cb117399e0d5\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") "
Oct 30 19:41:24.645182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.644738    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55866653-01c9-4307-9d74-cb117399e0d5-kube-api-access-bsq5d\" (UniqueName: \"kubernetes.io/projected/55866653-01c9-4307-9d74-cb117399e0d5-kube-api-access-bsq5d\") pod \"55866653-01c9-4307-9d74-cb117399e0d5\" (UID: \"55866653-01c9-4307-9d74-cb117399e0d5\") "
Oct 30 19:41:24.646696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.646517    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-tmp-dir" pod "55866653-01c9-4307-9d74-cb117399e0d5" (UID: "55866653-01c9-4307-9d74-cb117399e0d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:24.647386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.647022    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-home" pod "55866653-01c9-4307-9d74-cb117399e0d5" (UID: "55866653-01c9-4307-9d74-cb117399e0d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:24.647386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.647032    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-model-cache" pod "55866653-01c9-4307-9d74-cb117399e0d5" (UID: "55866653-01c9-4307-9d74-cb117399e0d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:24.650576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.650248    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-dshm" pod "55866653-01c9-4307-9d74-cb117399e0d5" (UID: "55866653-01c9-4307-9d74-cb117399e0d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:24.650576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.650335    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55866653-01c9-4307-9d74-cb117399e0d5-kube-api-access-bsq5d" pod "55866653-01c9-4307-9d74-cb117399e0d5" (UID: "55866653-01c9-4307-9d74-cb117399e0d5"). InnerVolumeSpecName "kube-api-access-bsq5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:24.662637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.661818    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55866653-01c9-4307-9d74-cb117399e0d5-tls-certs" pod "55866653-01c9-4307-9d74-cb117399e0d5" (UID: "55866653-01c9-4307-9d74-cb117399e0d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:24.748578 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.748535    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55866653-01c9-4307-9d74-cb117399e0d5-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:24.748578 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.748571    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:24.748578 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.748580    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:24.748859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.748592    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsq5d\" (UniqueName: \"kubernetes.io/projected/55866653-01c9-4307-9d74-cb117399e0d5-kube-api-access-bsq5d\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:24.748859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.748604    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:24.748859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:24.748615    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55866653-01c9-4307-9d74-cb117399e0d5-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:25.412450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:25.412402    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": context deadline exceeded"
Oct 30 19:41:25.553446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:25.553405    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"
Oct 30 19:41:25.585201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:25.585155    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"]
Oct 30 19:41:25.590121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:25.590081    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8e6762/auth-test-model-8e6762-kserve-6dd95d76b9-bg2g7"]
Oct 30 19:41:26.151668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:26.151631    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55866653-01c9-4307-9d74-cb117399e0d5" path="/var/lib/kubelet/pods/55866653-01c9-4307-9d74-cb117399e0d5/volumes"
Oct 30 19:41:27.579446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.579403    6777 generic.go:356] "Generic (PLEG): container finished" podID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerID="3f192d20da9794a057741959b0f5ad644e7f2978395fb42912837efaff0590ab" exitCode=0
Oct 30 19:41:27.579953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.579479    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" event={"ID":"d5d4251f-32c1-421b-aa93-79794a66b2ac","Type":"ContainerDied","Data":"3f192d20da9794a057741959b0f5ad644e7f2978395fb42912837efaff0590ab"}
Oct 30 19:41:27.775169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.775139    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:41:27.817285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.817248    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"]
Oct 30 19:41:27.818819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.818791    6777 cpu_manager.go:414] "containerMap: removing container" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main"
Oct 30 19:41:27.818980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.818968    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main"
Oct 30 19:41:27.819145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.819133    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy"
Oct 30 19:41:27.819228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.819218    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy"
Oct 30 19:41:27.819547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.819536    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="55866653-01c9-4307-9d74-cb117399e0d5" containerName="main"
Oct 30 19:41:27.819645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.819635    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" containerName="istio-proxy"
Oct 30 19:41:27.852975 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.852929    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:27.859201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.859166    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fd4577-7428-4f87-96f6-5e84954bbaa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7\" (UID: \"70fd4577-7428-4f87-96f6-5e84954bbaa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:27.859883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.859856    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4qf\" (UniqueName: \"kubernetes.io/projected/70fd4577-7428-4f87-96f6-5e84954bbaa2-kube-api-access-tz4qf\") pod \"payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7\" (UID: \"70fd4577-7428-4f87-96f6-5e84954bbaa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:27.861998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.861968    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"]
Oct 30 19:41:27.961973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.961884    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-token\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-token\") pod \"d5d4251f-32c1-421b-aa93-79794a66b2ac\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") "
Oct 30 19:41:27.961973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.961953    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-data\") pod \"d5d4251f-32c1-421b-aa93-79794a66b2ac\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") "
Oct 30 19:41:27.962349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.961997    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d4251f-32c1-421b-aa93-79794a66b2ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d4251f-32c1-421b-aa93-79794a66b2ac-istiod-ca-cert\") pod \"d5d4251f-32c1-421b-aa93-79794a66b2ac\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") "
Oct 30 19:41:27.962349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.962038    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-credential-socket\") pod \"d5d4251f-32c1-421b-aa93-79794a66b2ac\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") "
Oct 30 19:41:27.962349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.962117    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-certs\") pod \"d5d4251f-32c1-421b-aa93-79794a66b2ac\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") "
Oct 30 19:41:27.962349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.962248    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-envoy\") pod \"d5d4251f-32c1-421b-aa93-79794a66b2ac\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") "
Oct 30 19:41:27.962547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.962416    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-credential-socket" pod "d5d4251f-32c1-421b-aa93-79794a66b2ac" (UID: "d5d4251f-32c1-421b-aa93-79794a66b2ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:27.962547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.962396    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-data" pod "d5d4251f-32c1-421b-aa93-79794a66b2ac" (UID: "d5d4251f-32c1-421b-aa93-79794a66b2ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:27.962547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.962470    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-certs" pod "d5d4251f-32c1-421b-aa93-79794a66b2ac" (UID: "d5d4251f-32c1-421b-aa93-79794a66b2ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:27.962547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.962510    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d4251f-32c1-421b-aa93-79794a66b2ac-istiod-ca-cert" pod "d5d4251f-32c1-421b-aa93-79794a66b2ac" (UID: "d5d4251f-32c1-421b-aa93-79794a66b2ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:27.962752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.962733    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-socket\") pod \"d5d4251f-32c1-421b-aa93-79794a66b2ac\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") "
Oct 30 19:41:27.962907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.962885    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-kube-api-access-6r992\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-kube-api-access-6r992\") pod \"d5d4251f-32c1-421b-aa93-79794a66b2ac\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") "
Oct 30 19:41:27.963051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.963027    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-podinfo\") pod \"d5d4251f-32c1-421b-aa93-79794a66b2ac\" (UID: \"d5d4251f-32c1-421b-aa93-79794a66b2ac\") "
Oct 30 19:41:27.963197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.963026    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-socket" pod "d5d4251f-32c1-421b-aa93-79794a66b2ac" (UID: "d5d4251f-32c1-421b-aa93-79794a66b2ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:27.963476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.963455    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4qf\" (UniqueName: \"kubernetes.io/projected/70fd4577-7428-4f87-96f6-5e84954bbaa2-kube-api-access-tz4qf\") pod \"payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7\" (UID: \"70fd4577-7428-4f87-96f6-5e84954bbaa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:27.963907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.963886    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fd4577-7428-4f87-96f6-5e84954bbaa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7\" (UID: \"70fd4577-7428-4f87-96f6-5e84954bbaa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:27.964106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.964086    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d4251f-32c1-421b-aa93-79794a66b2ac-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:27.964236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.964111    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:27.964236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.964127    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:27.964236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.964143    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:27.964236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.964156    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:27.965023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.964843    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fd4577-7428-4f87-96f6-5e84954bbaa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7\" (UID: \"70fd4577-7428-4f87-96f6-5e84954bbaa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:27.965129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.965047    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-envoy" pod "d5d4251f-32c1-421b-aa93-79794a66b2ac" (UID: "d5d4251f-32c1-421b-aa93-79794a66b2ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:27.965253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.965191    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-token" pod "d5d4251f-32c1-421b-aa93-79794a66b2ac" (UID: "d5d4251f-32c1-421b-aa93-79794a66b2ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:27.965439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.965421    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-podinfo" pod "d5d4251f-32c1-421b-aa93-79794a66b2ac" (UID: "d5d4251f-32c1-421b-aa93-79794a66b2ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:41:27.966164 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.966140    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-kube-api-access-6r992" pod "d5d4251f-32c1-421b-aa93-79794a66b2ac" (UID: "d5d4251f-32c1-421b-aa93-79794a66b2ac"). InnerVolumeSpecName "kube-api-access-6r992". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:27.976668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:27.976630    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4qf\" (UniqueName: \"kubernetes.io/projected/70fd4577-7428-4f87-96f6-5e84954bbaa2-kube-api-access-tz4qf\") pod \"payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7\" (UID: \"70fd4577-7428-4f87-96f6-5e84954bbaa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:28.065805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.065756    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r992\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-kube-api-access-6r992\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.065805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.065799    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.065805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.065810    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.066768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.065820    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d4251f-32c1-421b-aa93-79794a66b2ac-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.183760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.183719    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:28.480236 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:41:28.480204    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fd4577_7428_4f87_96f6_5e84954bbaa2.slice/crio-1d89ef4554b05da58919a13c38eda27f8e9934dcc55d10ba1162e8fca60d7a80 WatchSource:0}: Error finding container 1d89ef4554b05da58919a13c38eda27f8e9934dcc55d10ba1162e8fca60d7a80: Status 404 returned error can't find the container with id 1d89ef4554b05da58919a13c38eda27f8e9934dcc55d10ba1162e8fca60d7a80
Oct 30 19:41:28.481236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.481212    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"]
Oct 30 19:41:28.591299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.591252    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7" event={"ID":"70fd4577-7428-4f87-96f6-5e84954bbaa2","Type":"ContainerStarted","Data":"749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc"}
Oct 30 19:41:28.591819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.591328    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7" event={"ID":"70fd4577-7428-4f87-96f6-5e84954bbaa2","Type":"ContainerStarted","Data":"1d89ef4554b05da58919a13c38eda27f8e9934dcc55d10ba1162e8fca60d7a80"}
Oct 30 19:41:28.591819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.591360    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:28.593025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.593003    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"
Oct 30 19:41:28.593216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.593022    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s" event={"ID":"d5d4251f-32c1-421b-aa93-79794a66b2ac","Type":"ContainerDied","Data":"c1bd8ecc7e51bf9d77f43753c62d75aeb5b8fc6f8f0f91b80c14f0ceb78dfbd7"}
Oct 30 19:41:28.593216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.593140    6777 scope.go:122] "RemoveContainer" containerID="3f192d20da9794a057741959b0f5ad644e7f2978395fb42912837efaff0590ab"
Oct 30 19:41:28.616872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.616810    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7" podStartSLOduration=7.616791976 podStartE2EDuration="7.616791976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:28.612907873 +0000 UTC m=+2070.609823257" watchObservedRunningTime="2026-10-30 19:41:28.616791976 +0000 UTC m=+2070.613707361"
Oct 30 19:41:28.640769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.640725    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"]
Oct 30 19:41:28.646119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:28.646052    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-2tc5s"]
Oct 30 19:41:30.117124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:30.117085    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d4251f-32c1-421b-aa93-79794a66b2ac" path="/var/lib/kubelet/pods/d5d4251f-32c1-421b-aa93-79794a66b2ac/volumes"
Oct 30 19:41:33.071634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.071593    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"]
Oct 30 19:41:33.078120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.078095    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:33.087877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.087846    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e50648\""
Oct 30 19:41:33.088186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.087998    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e50648-dockercfg-7txtz\""
Oct 30 19:41:33.095251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.095223    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"]
Oct 30 19:41:33.117508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.117462    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9\" (UID: \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:33.117727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.117647    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gjg\" (UniqueName: \"kubernetes.io/projected/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-kube-api-access-89gjg\") pod \"payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9\" (UID: \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:33.271088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.248487    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9\" (UID: \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:33.271088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.248617    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gjg\" (UniqueName: \"kubernetes.io/projected/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-kube-api-access-89gjg\") pod \"payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9\" (UID: \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:33.279094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.271769    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9\" (UID: \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:33.338144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.329805    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gjg\" (UniqueName: \"kubernetes.io/projected/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-kube-api-access-89gjg\") pod \"payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9\" (UID: \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:33.403451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.403404    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:33.878576 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:41:33.878545    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4feba2ef_8e08_4c91_87f5_b3c3ea3115d9.slice/crio-698a9f7145536f9a88a79bcd8e18777b996f5f3b5ee75823dd9ba85619574ca0 WatchSource:0}: Error finding container 698a9f7145536f9a88a79bcd8e18777b996f5f3b5ee75823dd9ba85619574ca0: Status 404 returned error can't find the container with id 698a9f7145536f9a88a79bcd8e18777b996f5f3b5ee75823dd9ba85619574ca0
Oct 30 19:41:33.878765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:33.878736    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"]
Oct 30 19:41:34.547602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:34.547569    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:41:34.678230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:34.678196    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9" event={"ID":"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9","Type":"ContainerStarted","Data":"7170abad143c00e586e0d806e62998f84d7f186eb050e04c56ba10121eb6764c"}
Oct 30 19:41:34.678427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:34.678250    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9" event={"ID":"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9","Type":"ContainerStarted","Data":"698a9f7145536f9a88a79bcd8e18777b996f5f3b5ee75823dd9ba85619574ca0"}
Oct 30 19:41:34.678427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:34.678301    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:34.705207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:34.705153    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9" podStartSLOduration=1.705138046 podStartE2EDuration="1.705138046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:34.702331877 +0000 UTC m=+2076.699247262" watchObservedRunningTime="2026-10-30 19:41:34.705138046 +0000 UTC m=+2076.702053435"
Oct 30 19:41:39.608573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:39.608483    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:41:43.798085 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:41:43.790395    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:41:44.243220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:44.231164    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:44.313210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:44.313162    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:44.339085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:44.327627    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Oct 30 19:41:44.339085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:44.338707    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:44.390095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:44.369313    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:41:44.881100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:44.877143    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.262147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.244162    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 19:41:45.262147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.244786    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:41:45.264287 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.263193    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.322086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.320650    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.236:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.342381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.329238    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:41:45.342381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.330122    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:41:45.343678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.343633    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.369085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.344408    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:41:45.552075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.452231    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.552401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.469157    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.552753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.469196    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": context deadline exceeded"
Oct 30 19:41:45.552966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.469217    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:41:45.588096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.469240    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:41:45.642086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.565174    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:41:45.642461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.642432    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:41:45.649906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.591288    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.234:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.836791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.834671    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:41:45.837344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.837294    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:41:45.884766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.884717    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.903666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:45.886239    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:41:46.292127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:46.292096    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:41:46.301443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:46.301321    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:41:46.323654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:41:46.321275    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28"
Oct 30 19:42:01.527006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:01.526961    6777 scope.go:122] "RemoveContainer" containerID="c3edf7eaa94c3100f3ad4fa7de8124da2d09de7031e7c8574c70e38517b2d6ca"
Oct 30 19:42:01.540012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:01.539982    6777 scope.go:122] "RemoveContainer" containerID="eac0b33024879d42f748ae091d8a5e7231017dc3a1c3ba1484447285fd86b0c6"
Oct 30 19:42:09.035953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:09.035914    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"]
Oct 30 19:42:09.040823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:09.040792    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fd39f4-sgch6"]
Oct 30 19:42:10.116547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:10.116510    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f30eef-03fc-46a7-957a-c9ea17d1f1af" path="/var/lib/kubelet/pods/90f30eef-03fc-46a7-957a-c9ea17d1f1af/volumes"
Oct 30 19:42:13.643156 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:42:13.643113    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:42:21.878052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:21.878004    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:42:24.226091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.225171    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.255783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.242163    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:42:24.255783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.242593    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.236:8000/ready\": context deadline exceeded"
Oct 30 19:42:24.328679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.328141    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.339703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.339476    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.404041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.399164    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.404041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.399596    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:42:24.404041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.400166    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.404041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.401725    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.404041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.401984    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": context deadline exceeded"
Oct 30 19:42:24.404041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.402270    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:42:24.709893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:24.707613    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.234:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:35.274069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:35.274012    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"]
Oct 30 19:42:43.643931 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:42:43.643889    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:42:45.894349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:45.894310    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"]
Oct 30 19:42:45.899571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:45.899544    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:45.909520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:45.909476    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"]
Oct 30 19:42:45.915672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:45.915642    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:45.915861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:45.915780    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:42:46.002423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.002381    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mfh\" (UniqueName: \"kubernetes.io/projected/7b91e2d0-dcea-496f-80af-a76992292405-kube-api-access-s8mfh\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.002684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.002640    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.002755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.002692    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.104889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.104849    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mfh\" (UniqueName: \"kubernetes.io/projected/7b91e2d0-dcea-496f-80af-a76992292405-kube-api-access-s8mfh\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.105147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.105032    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.105147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.105128    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.105870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.105841    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.107740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.107713    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.118630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.118600    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mfh\" (UniqueName: \"kubernetes.io/projected/7b91e2d0-dcea-496f-80af-a76992292405-kube-api-access-s8mfh\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.220253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.220150    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:46.541290 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:42:46.541247    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b91e2d0_dcea_496f_80af_a76992292405.slice/crio-c41aebfc775071ccd0dcaeeca678b0445d3a32af4866bcd251d0ef7b92953991 WatchSource:0}: Error finding container c41aebfc775071ccd0dcaeeca678b0445d3a32af4866bcd251d0ef7b92953991: Status 404 returned error can't find the container with id c41aebfc775071ccd0dcaeeca678b0445d3a32af4866bcd251d0ef7b92953991
Oct 30 19:42:46.545039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:46.545004    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"]
Oct 30 19:42:47.201874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:47.201773    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc" event={"ID":"7b91e2d0-dcea-496f-80af-a76992292405","Type":"ContainerStarted","Data":"5ecd70bd2b17f039dfdb201471302cf2818a86d51b32d3b7140f05e7f14fd2a6"}
Oct 30 19:42:47.201874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:47.201840    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc" event={"ID":"7b91e2d0-dcea-496f-80af-a76992292405","Type":"ContainerStarted","Data":"c41aebfc775071ccd0dcaeeca678b0445d3a32af4866bcd251d0ef7b92953991"}
Oct 30 19:42:47.272114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:47.272040    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc" podStartSLOduration=2.272025519 podStartE2EDuration="2.272025519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:47.26972378 +0000 UTC m=+2149.266639164" watchObservedRunningTime="2026-10-30 19:42:47.272025519 +0000 UTC m=+2149.268940907"
Oct 30 19:42:48.213134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:48.213092    6777 generic.go:356] "Generic (PLEG): container finished" podID="7b91e2d0-dcea-496f-80af-a76992292405" containerID="5ecd70bd2b17f039dfdb201471302cf2818a86d51b32d3b7140f05e7f14fd2a6" exitCode=0
Oct 30 19:42:48.214160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:48.213571    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc" event={"ID":"7b91e2d0-dcea-496f-80af-a76992292405","Type":"ContainerDied","Data":"5ecd70bd2b17f039dfdb201471302cf2818a86d51b32d3b7140f05e7f14fd2a6"}
Oct 30 19:42:49.480624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.480548    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:49.599663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.599599    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b91e2d0-dcea-496f-80af-a76992292405-kube-api-access-s8mfh\" (UniqueName: \"kubernetes.io/projected/7b91e2d0-dcea-496f-80af-a76992292405-kube-api-access-s8mfh\") pod \"7b91e2d0-dcea-496f-80af-a76992292405\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") "
Oct 30 19:42:49.599936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.599793    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-openshift-ca\") pod \"7b91e2d0-dcea-496f-80af-a76992292405\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") "
Oct 30 19:42:49.599936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.599913    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-xks-ca\") pod \"7b91e2d0-dcea-496f-80af-a76992292405\" (UID: \"7b91e2d0-dcea-496f-80af-a76992292405\") "
Oct 30 19:42:49.600584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.600503    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-openshift-ca" pod "7b91e2d0-dcea-496f-80af-a76992292405" (UID: "7b91e2d0-dcea-496f-80af-a76992292405"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:49.601199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.600953    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:49.602883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.602846    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-xks-ca" pod "7b91e2d0-dcea-496f-80af-a76992292405" (UID: "7b91e2d0-dcea-496f-80af-a76992292405"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:49.604355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.604327    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b91e2d0-dcea-496f-80af-a76992292405-kube-api-access-s8mfh" pod "7b91e2d0-dcea-496f-80af-a76992292405" (UID: "7b91e2d0-dcea-496f-80af-a76992292405"). InnerVolumeSpecName "kube-api-access-s8mfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:49.703924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.703877    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8mfh\" (UniqueName: \"kubernetes.io/projected/7b91e2d0-dcea-496f-80af-a76992292405-kube-api-access-s8mfh\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:49.703924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:49.703922    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e2d0-dcea-496f-80af-a76992292405-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:50.237620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.237590    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"
Oct 30 19:42:50.237835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.237616    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc" event={"ID":"7b91e2d0-dcea-496f-80af-a76992292405","Type":"ContainerDied","Data":"c41aebfc775071ccd0dcaeeca678b0445d3a32af4866bcd251d0ef7b92953991"}
Oct 30 19:42:50.237835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.237678    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41aebfc775071ccd0dcaeeca678b0445d3a32af4866bcd251d0ef7b92953991"
Oct 30 19:42:50.297322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.297278    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" containerID="cri-o://73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49" gracePeriod=45
Oct 30 19:42:50.380019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.379911    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.227:8000/ready\": dial tcp 10.134.0.227:8000: connect: connection refused"
Oct 30 19:42:50.611399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.611373    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:42:50.622935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.622903    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-home\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-home\") pod \"d89e2513-3b72-4674-bee1-ed6632d75c2b\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") "
Oct 30 19:42:50.623147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.623000    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-tmp-dir\") pod \"d89e2513-3b72-4674-bee1-ed6632d75c2b\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") "
Oct 30 19:42:50.623216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.623188    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-dshm\") pod \"d89e2513-3b72-4674-bee1-ed6632d75c2b\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") "
Oct 30 19:42:50.623291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.623265    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-home" pod "d89e2513-3b72-4674-bee1-ed6632d75c2b" (UID: "d89e2513-3b72-4674-bee1-ed6632d75c2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:50.623370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.623273    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-tmp-dir" pod "d89e2513-3b72-4674-bee1-ed6632d75c2b" (UID: "d89e2513-3b72-4674-bee1-ed6632d75c2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:50.623438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.623382    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89e2513-3b72-4674-bee1-ed6632d75c2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d89e2513-3b72-4674-bee1-ed6632d75c2b-tls-certs\") pod \"d89e2513-3b72-4674-bee1-ed6632d75c2b\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") "
Oct 30 19:42:50.623438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.623417    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89e2513-3b72-4674-bee1-ed6632d75c2b-kube-api-access-nt5sw\" (UniqueName: \"kubernetes.io/projected/d89e2513-3b72-4674-bee1-ed6632d75c2b-kube-api-access-nt5sw\") pod \"d89e2513-3b72-4674-bee1-ed6632d75c2b\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") "
Oct 30 19:42:50.623551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.623480    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-model-cache\") pod \"d89e2513-3b72-4674-bee1-ed6632d75c2b\" (UID: \"d89e2513-3b72-4674-bee1-ed6632d75c2b\") "
Oct 30 19:42:50.623815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.623793    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-model-cache" pod "d89e2513-3b72-4674-bee1-ed6632d75c2b" (UID: "d89e2513-3b72-4674-bee1-ed6632d75c2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:50.624366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.624341    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:50.624477 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.624370    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:50.624477 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.624387    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:50.625596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.625560    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-dshm" pod "d89e2513-3b72-4674-bee1-ed6632d75c2b" (UID: "d89e2513-3b72-4674-bee1-ed6632d75c2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:50.625685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.625647    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89e2513-3b72-4674-bee1-ed6632d75c2b-tls-certs" pod "d89e2513-3b72-4674-bee1-ed6632d75c2b" (UID: "d89e2513-3b72-4674-bee1-ed6632d75c2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:50.625799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.625732    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89e2513-3b72-4674-bee1-ed6632d75c2b-kube-api-access-nt5sw" pod "d89e2513-3b72-4674-bee1-ed6632d75c2b" (UID: "d89e2513-3b72-4674-bee1-ed6632d75c2b"). InnerVolumeSpecName "kube-api-access-nt5sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:50.726391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.726348    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89e2513-3b72-4674-bee1-ed6632d75c2b-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:50.726391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.726389    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89e2513-3b72-4674-bee1-ed6632d75c2b-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:50.726621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:50.726406    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt5sw\" (UniqueName: \"kubernetes.io/projected/d89e2513-3b72-4674-bee1-ed6632d75c2b-kube-api-access-nt5sw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:51.248040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:51.248000    6777 generic.go:356] "Generic (PLEG): container finished" podID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerID="73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49" exitCode=0
Oct 30 19:42:51.248040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:51.248075    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"
Oct 30 19:42:51.248335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:51.248045    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" event={"ID":"d89e2513-3b72-4674-bee1-ed6632d75c2b","Type":"ContainerDied","Data":"73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49"}
Oct 30 19:42:51.248335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:51.248123    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw" event={"ID":"d89e2513-3b72-4674-bee1-ed6632d75c2b","Type":"ContainerDied","Data":"2a2b2dc6b9345e883642d5857a18c08dfa1bd2412f1a22099c5d50a26caa697f"}
Oct 30 19:42:51.248335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:51.248142    6777 scope.go:122] "RemoveContainer" containerID="73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49"
Oct 30 19:42:51.262528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:51.262506    6777 scope.go:122] "RemoveContainer" containerID="73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49"
Oct 30 19:42:51.262878 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:42:51.262851    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49\": container with ID starting with 73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49 not found: ID does not exist" containerID="73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49"
Oct 30 19:42:51.262968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:51.262887    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49"} err="failed to get container status \"73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49\": rpc error: code = NotFound desc = could not find container \"73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49\": container with ID starting with 73d44a29dd1bf544cee4750e1b813b23b567fac0c9a1c60b7563e774ac075a49 not found: ID does not exist"
Oct 30 19:42:51.276128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:51.276087    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"]
Oct 30 19:42:51.282315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:51.282282    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-69c573/sub-test-model-69c573-kserve-7f49d7d89f-7fhdw"]
Oct 30 19:42:52.117386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:52.117349    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" path="/var/lib/kubelet/pods/d89e2513-3b72-4674-bee1-ed6632d75c2b/volumes"
Oct 30 19:42:56.046957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:56.046918    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"]
Oct 30 19:42:56.060647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:42:56.060609    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"]
Oct 30 19:43:01.047708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:01.047652    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s" podUID="09f8ec32-8902-466d-89d2-bb1b83baf1f6" containerName="payload-processing" containerID="cri-o://a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c" gracePeriod=25
Oct 30 19:43:01.061298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:01.061250    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt" podUID="ecabcbae-0ee0-45f9-8b89-54052c9a5a65" containerName="payload-pre-processing" containerID="cri-o://9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b" gracePeriod=25
Oct 30 19:43:03.292715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:03.292665    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt" podUID="ecabcbae-0ee0-45f9-8b89-54052c9a5a65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:04.353234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:04.353180    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s" podUID="09f8ec32-8902-466d-89d2-bb1b83baf1f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:05.230212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:05.230160    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:43:05.232565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:05.232528    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.236:8000/ready\": context deadline exceeded"
Oct 30 19:43:05.369389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:05.369341    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:43:05.391255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:05.391202    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:13.291927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:13.291837    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt" podUID="ecabcbae-0ee0-45f9-8b89-54052c9a5a65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:13.643969 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:13.643872    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:43:14.275083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:14.274321    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"]
Oct 30 19:43:14.294015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:14.293962    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-69c573-n65hc"]
Oct 30 19:43:14.320026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:14.319982    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s" podUID="09f8ec32-8902-466d-89d2-bb1b83baf1f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:16.116632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:16.116592    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b91e2d0-dcea-496f-80af-a76992292405" path="/var/lib/kubelet/pods/7b91e2d0-dcea-496f-80af-a76992292405/volumes"
Oct 30 19:43:20.308103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.308051    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"]
Oct 30 19:43:20.309165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.308989    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7b91e2d0-dcea-496f-80af-a76992292405" containerName="revoke-keys"
Oct 30 19:43:20.309165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.309003    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b91e2d0-dcea-496f-80af-a76992292405" containerName="revoke-keys"
Oct 30 19:43:20.309165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.309025    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main"
Oct 30 19:43:20.309165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.309030    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main"
Oct 30 19:43:20.309400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.309179    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b91e2d0-dcea-496f-80af-a76992292405" containerName="revoke-keys"
Oct 30 19:43:20.309400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.309190    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89e2513-3b72-4674-bee1-ed6632d75c2b" containerName="main"
Oct 30 19:43:20.313926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.313895    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.318304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.318274    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:20.319777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.319753    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:43:20.322698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.322673    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"]
Oct 30 19:43:20.419677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.419637    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c87bba-3998-433f-b361-b232946ace9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.419902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.419783    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c87bba-3998-433f-b361-b232946ace9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.419962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.419947    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlcl2\" (UniqueName: \"kubernetes.io/projected/68c87bba-3998-433f-b361-b232946ace9e-kube-api-access-xlcl2\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.522005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.521963    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c87bba-3998-433f-b361-b232946ace9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.522248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.522116    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c87bba-3998-433f-b361-b232946ace9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.522248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.522235    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlcl2\" (UniqueName: \"kubernetes.io/projected/68c87bba-3998-433f-b361-b232946ace9e-kube-api-access-xlcl2\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.522824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.522798    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c87bba-3998-433f-b361-b232946ace9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.524629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.524609    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c87bba-3998-433f-b361-b232946ace9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.535098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.535041    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlcl2\" (UniqueName: \"kubernetes.io/projected/68c87bba-3998-433f-b361-b232946ace9e-kube-api-access-xlcl2\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.631480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.631387    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:20.940232 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:43:20.940200    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c87bba_3998_433f_b361_b232946ace9e.slice/crio-e7cafe83b125ca12cfca97c9dff4af50b439f3a0ea345c605679d2e01a30f884 WatchSource:0}: Error finding container e7cafe83b125ca12cfca97c9dff4af50b439f3a0ea345c605679d2e01a30f884: Status 404 returned error can't find the container with id e7cafe83b125ca12cfca97c9dff4af50b439f3a0ea345c605679d2e01a30f884
Oct 30 19:43:20.941599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:20.941568    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"]
Oct 30 19:43:21.462301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.462271    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:43:21.543848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.543813    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f8ec32-8902-466d-89d2-bb1b83baf1f6-kube-api-access-56rs9\" (UniqueName: \"kubernetes.io/projected/09f8ec32-8902-466d-89d2-bb1b83baf1f6-kube-api-access-56rs9\") pod \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\" (UID: \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\") "
Oct 30 19:43:21.544194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.544167    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f8ec32-8902-466d-89d2-bb1b83baf1f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f8ec32-8902-466d-89d2-bb1b83baf1f6-plugins-config-volume\") pod \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\" (UID: \"09f8ec32-8902-466d-89d2-bb1b83baf1f6\") "
Oct 30 19:43:21.545052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.545020    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f8ec32-8902-466d-89d2-bb1b83baf1f6-plugins-config-volume" pod "09f8ec32-8902-466d-89d2-bb1b83baf1f6" (UID: "09f8ec32-8902-466d-89d2-bb1b83baf1f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:21.545471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.545435    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f8ec32-8902-466d-89d2-bb1b83baf1f6-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:21.546631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.546606    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f8ec32-8902-466d-89d2-bb1b83baf1f6-kube-api-access-56rs9" pod "09f8ec32-8902-466d-89d2-bb1b83baf1f6" (UID: "09f8ec32-8902-466d-89d2-bb1b83baf1f6"). InnerVolumeSpecName "kube-api-access-56rs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:21.582264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.582231    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:43:21.647174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.647135    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-kube-api-access-jhdh6\" (UniqueName: \"kubernetes.io/projected/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-kube-api-access-jhdh6\") pod \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\" (UID: \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\") "
Oct 30 19:43:21.647434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.647416    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-plugins-config-volume\") pod \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\" (UID: \"ecabcbae-0ee0-45f9-8b89-54052c9a5a65\") "
Oct 30 19:43:21.647885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.647854    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-plugins-config-volume" pod "ecabcbae-0ee0-45f9-8b89-54052c9a5a65" (UID: "ecabcbae-0ee0-45f9-8b89-54052c9a5a65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:21.648384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.648362    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rs9\" (UniqueName: \"kubernetes.io/projected/09f8ec32-8902-466d-89d2-bb1b83baf1f6-kube-api-access-56rs9\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:21.648467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.648390    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:21.650967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.650937    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-kube-api-access-jhdh6" pod "ecabcbae-0ee0-45f9-8b89-54052c9a5a65" (UID: "ecabcbae-0ee0-45f9-8b89-54052c9a5a65"). InnerVolumeSpecName "kube-api-access-jhdh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:21.686747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.686704    6777 generic.go:356] "Generic (PLEG): container finished" podID="09f8ec32-8902-466d-89d2-bb1b83baf1f6" containerID="a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c" exitCode=0
Oct 30 19:43:21.686974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.686787    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"
Oct 30 19:43:21.686974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.686789    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s" event={"ID":"09f8ec32-8902-466d-89d2-bb1b83baf1f6","Type":"ContainerDied","Data":"a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c"}
Oct 30 19:43:21.686974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.686954    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s" event={"ID":"09f8ec32-8902-466d-89d2-bb1b83baf1f6","Type":"ContainerDied","Data":"0fbc95182c0a4b42f901b3c860adcedeac190584f78692e685213147198b9772"}
Oct 30 19:43:21.687222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.686998    6777 scope.go:122] "RemoveContainer" containerID="a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c"
Oct 30 19:43:21.688953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.688924    6777 generic.go:356] "Generic (PLEG): container finished" podID="ecabcbae-0ee0-45f9-8b89-54052c9a5a65" containerID="9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b" exitCode=0
Oct 30 19:43:21.689142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.688963    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt" event={"ID":"ecabcbae-0ee0-45f9-8b89-54052c9a5a65","Type":"ContainerDied","Data":"9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b"}
Oct 30 19:43:21.689142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.688985    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"
Oct 30 19:43:21.689142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.689037    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt" event={"ID":"ecabcbae-0ee0-45f9-8b89-54052c9a5a65","Type":"ContainerDied","Data":"b902815daf0feec7c114be3aed08816b22ef1363b11b55bbed24ef994dd22d10"}
Oct 30 19:43:21.694647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.694620    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" event={"ID":"68c87bba-3998-433f-b361-b232946ace9e","Type":"ContainerStarted","Data":"1edfc31668a838c2320d228849d4cd6c0467a175496ca7d65092a0e15d8a73de"}
Oct 30 19:43:21.694803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.694675    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" event={"ID":"68c87bba-3998-433f-b361-b232946ace9e","Type":"ContainerStarted","Data":"e7cafe83b125ca12cfca97c9dff4af50b439f3a0ea345c605679d2e01a30f884"}
Oct 30 19:43:21.703130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.703022    6777 scope.go:122] "RemoveContainer" containerID="a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c"
Oct 30 19:43:21.703447 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:21.703418    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c\": container with ID starting with a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c not found: ID does not exist" containerID="a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c"
Oct 30 19:43:21.703546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.703459    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c"} err="failed to get container status \"a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c\": rpc error: code = NotFound desc = could not find container \"a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c\": container with ID starting with a457e8e2f34d8a2d9a01300bf1925c4da06e9f0a7d144c429afce0517cb74f9c not found: ID does not exist"
Oct 30 19:43:21.703546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.703484    6777 scope.go:122] "RemoveContainer" containerID="9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b"
Oct 30 19:43:21.716005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.715975    6777 scope.go:122] "RemoveContainer" containerID="9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b"
Oct 30 19:43:21.716385 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:21.716364    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b\": container with ID starting with 9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b not found: ID does not exist" containerID="9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b"
Oct 30 19:43:21.716442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.716396    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b"} err="failed to get container status \"9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b\": rpc error: code = NotFound desc = could not find container \"9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b\": container with ID starting with 9742ae2135608b8ddaa3c515dddbd705fd3f99ae5e04a8225177ea821f49f14b not found: ID does not exist"
Oct 30 19:43:21.724485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.724433    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" podStartSLOduration=1.724416958 podStartE2EDuration="1.724416958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:21.71915989 +0000 UTC m=+2183.716075276" watchObservedRunningTime="2026-10-30 19:43:21.724416958 +0000 UTC m=+2183.721332342"
Oct 30 19:43:21.741574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.741542    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"]
Oct 30 19:43:21.746617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.746581    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-69c573-868f49556c-xjzkt"]
Oct 30 19:43:21.750519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.750484    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhdh6\" (UniqueName: \"kubernetes.io/projected/ecabcbae-0ee0-45f9-8b89-54052c9a5a65-kube-api-access-jhdh6\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:21.760340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.760307    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"]
Oct 30 19:43:21.765307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:21.765275    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-69c573-689db57bf8-9hf4s"]
Oct 30 19:43:22.175200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:22.172799    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f8ec32-8902-466d-89d2-bb1b83baf1f6" path="/var/lib/kubelet/pods/09f8ec32-8902-466d-89d2-bb1b83baf1f6/volumes"
Oct 30 19:43:22.175200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:22.173338    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecabcbae-0ee0-45f9-8b89-54052c9a5a65" path="/var/lib/kubelet/pods/ecabcbae-0ee0-45f9-8b89-54052c9a5a65/volumes"
Oct 30 19:43:22.707259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:22.707223    6777 generic.go:356] "Generic (PLEG): container finished" podID="68c87bba-3998-433f-b361-b232946ace9e" containerID="1edfc31668a838c2320d228849d4cd6c0467a175496ca7d65092a0e15d8a73de" exitCode=7
Oct 30 19:43:22.707692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:22.707311    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" event={"ID":"68c87bba-3998-433f-b361-b232946ace9e","Type":"ContainerDied","Data":"1edfc31668a838c2320d228849d4cd6c0467a175496ca7d65092a0e15d8a73de"}
Oct 30 19:43:22.707738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:22.707710    6777 scope.go:122] "RemoveContainer" containerID="1edfc31668a838c2320d228849d4cd6c0467a175496ca7d65092a0e15d8a73de"
Oct 30 19:43:23.006130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:23.006090    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"]
Oct 30 19:43:23.006468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:23.006427    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy" containerID="cri-o://392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d" gracePeriod=30
Oct 30 19:43:23.017026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:23.016994    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"]
Oct 30 19:43:23.017275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:23.017205    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy" containerID="cri-o://a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee" gracePeriod=30
Oct 30 19:43:23.036276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:23.036239    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"]
Oct 30 19:43:23.036605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:23.036518    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy" containerID="cri-o://3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7" gracePeriod=30
Oct 30 19:43:23.540508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:23.540448    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:43:23.540688 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:23.540551    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:43:23.750546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:23.750443    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" event={"ID":"68c87bba-3998-433f-b361-b232946ace9e","Type":"ContainerStarted","Data":"ebeacfe26bb0e4c368232d869be6cbaa44b98e41971a65536fd695d083637a59"}
Oct 30 19:43:24.772549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:24.772514    6777 generic.go:356] "Generic (PLEG): container finished" podID="68c87bba-3998-433f-b361-b232946ace9e" containerID="ebeacfe26bb0e4c368232d869be6cbaa44b98e41971a65536fd695d083637a59" exitCode=7
Oct 30 19:43:24.773002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:24.772607    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" event={"ID":"68c87bba-3998-433f-b361-b232946ace9e","Type":"ContainerDied","Data":"ebeacfe26bb0e4c368232d869be6cbaa44b98e41971a65536fd695d083637a59"}
Oct 30 19:43:24.773002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:24.772708    6777 scope.go:122] "RemoveContainer" containerID="1edfc31668a838c2320d228849d4cd6c0467a175496ca7d65092a0e15d8a73de"
Oct 30 19:43:24.773002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:24.772956    6777 scope.go:122] "RemoveContainer" containerID="ebeacfe26bb0e4c368232d869be6cbaa44b98e41971a65536fd695d083637a59"
Oct 30 19:43:24.773403 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:24.773377    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf_odh-ai-gateway-infra(68c87bba-3998-433f-b361-b232946ace9e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" podUID="68c87bba-3998-433f-b361-b232946ace9e"
Oct 30 19:43:25.786677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:25.786647    6777 scope.go:122] "RemoveContainer" containerID="ebeacfe26bb0e4c368232d869be6cbaa44b98e41971a65536fd695d083637a59"
Oct 30 19:43:25.787215 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:25.787190    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf_odh-ai-gateway-infra(68c87bba-3998-433f-b361-b232946ace9e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" podUID="68c87bba-3998-433f-b361-b232946ace9e"
Oct 30 19:43:28.401136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.389004    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:43:28.440939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.440904    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"]
Oct 30 19:43:28.441592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.441565    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:43:28.442285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442260    6777 cpu_manager.go:414] "containerMap: removing container" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy"
Oct 30 19:43:28.442388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442288    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy"
Oct 30 19:43:28.442388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442325    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ecabcbae-0ee0-45f9-8b89-54052c9a5a65" containerName="payload-pre-processing"
Oct 30 19:43:28.442388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442335    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecabcbae-0ee0-45f9-8b89-54052c9a5a65" containerName="payload-pre-processing"
Oct 30 19:43:28.442388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442357    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09f8ec32-8902-466d-89d2-bb1b83baf1f6" containerName="payload-processing"
Oct 30 19:43:28.442388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442366    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f8ec32-8902-466d-89d2-bb1b83baf1f6" containerName="payload-processing"
Oct 30 19:43:28.442645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442411    6777 cpu_manager.go:414] "containerMap: removing container" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy"
Oct 30 19:43:28.442645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442419    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy"
Oct 30 19:43:28.442645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442458    6777 cpu_manager.go:414] "containerMap: removing container" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy"
Oct 30 19:43:28.442645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442466    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy"
Oct 30 19:43:28.442645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442638    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy"
Oct 30 19:43:28.442897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442656    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerName="istio-proxy"
Oct 30 19:43:28.442897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442672    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecabcbae-0ee0-45f9-8b89-54052c9a5a65" containerName="payload-pre-processing"
Oct 30 19:43:28.442897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.442684    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f8ec32-8902-466d-89d2-bb1b83baf1f6" containerName="payload-processing"
Oct 30 19:43:28.448518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.448485    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.461233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.461198    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"]
Oct 30 19:43:28.484909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.484880    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:43:28.486380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.485761    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"]
Oct 30 19:43:28.487954 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.487925    6777 cpu_manager.go:414] "containerMap: removing container" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy"
Oct 30 19:43:28.488168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.488130    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy"
Oct 30 19:43:28.488496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.488477    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerName="istio-proxy"
Oct 30 19:43:28.488629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.488501    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" containerName="istio-proxy"
Oct 30 19:43:28.491262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.490821    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/132771fd-242f-43c0-9c51-32dc14c71f98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132771fd-242f-43c0-9c51-32dc14c71f98-istio-podinfo\") pod \"132771fd-242f-43c0-9c51-32dc14c71f98\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") "
Oct 30 19:43:28.491262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.490926    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-credential-socket\") pod \"132771fd-242f-43c0-9c51-32dc14c71f98\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") "
Oct 30 19:43:28.493618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.493587    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.491014    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-socket\") pod \"132771fd-242f-43c0-9c51-32dc14c71f98\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") "
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.494526    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-kube-api-access-hdxqf\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-kube-api-access-hdxqf\") pod \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") "
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.494610    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-data\") pod \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") "
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.494684    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-podinfo\") pod \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") "
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.494815    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-istio-token\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-istio-token\") pod \"132771fd-242f-43c0-9c51-32dc14c71f98\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") "
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.494882    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-socket\") pod \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") "
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.494934    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istiod-ca-cert\") pod \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") "
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.494963    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/132771fd-242f-43c0-9c51-32dc14c71f98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132771fd-242f-43c0-9c51-32dc14c71f98-istiod-ca-cert\") pod \"132771fd-242f-43c0-9c51-32dc14c71f98\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") "
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.495025    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-kube-api-access-55zgm\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-kube-api-access-55zgm\") pod \"132771fd-242f-43c0-9c51-32dc14c71f98\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") "
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.495641    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istiod-ca-cert" pod "2dd96ad7-71c6-4389-bf36-dca4475cb5e5" (UID: "2dd96ad7-71c6-4389-bf36-dca4475cb5e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.495849    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-credential-socket" pod "132771fd-242f-43c0-9c51-32dc14c71f98" (UID: "132771fd-242f-43c0-9c51-32dc14c71f98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.496141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.495958    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-socket" pod "2dd96ad7-71c6-4389-bf36-dca4475cb5e5" (UID: "2dd96ad7-71c6-4389-bf36-dca4475cb5e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.496882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.496597    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132771fd-242f-43c0-9c51-32dc14c71f98-istiod-ca-cert" pod "132771fd-242f-43c0-9c51-32dc14c71f98" (UID: "132771fd-242f-43c0-9c51-32dc14c71f98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:28.497908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.497015    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-socket" pod "132771fd-242f-43c0-9c51-32dc14c71f98" (UID: "132771fd-242f-43c0-9c51-32dc14c71f98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.497908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.497038    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-data\") pod \"132771fd-242f-43c0-9c51-32dc14c71f98\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") "
Oct 30 19:43:28.497908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.497102    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-credential-socket\") pod \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") "
Oct 30 19:43:28.497908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.497140    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-token\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-token\") pod \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") "
Oct 30 19:43:28.497908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.497345    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-certs\") pod \"132771fd-242f-43c0-9c51-32dc14c71f98\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") "
Oct 30 19:43:28.497908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.497403    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-envoy\") pod \"132771fd-242f-43c0-9c51-32dc14c71f98\" (UID: \"132771fd-242f-43c0-9c51-32dc14c71f98\") "
Oct 30 19:43:28.497908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.497453    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-envoy\") pod \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") "
Oct 30 19:43:28.497908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.497482    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-certs\") pod \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\" (UID: \"2dd96ad7-71c6-4389-bf36-dca4475cb5e5\") "
Oct 30 19:43:28.513155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.512238    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-data" pod "2dd96ad7-71c6-4389-bf36-dca4475cb5e5" (UID: "2dd96ad7-71c6-4389-bf36-dca4475cb5e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.513155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.512464    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.513155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.512574    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.513155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.512618    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.513155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.512814    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.513155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.512920    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.513155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.512997    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jft\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-kube-api-access-m2jft\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.513155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.513034    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e94e090-5bce-47cf-9c75-a776e08b3265-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.513759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.513168    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.513759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.513235    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.513759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.513564    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.513759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.513586    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.513759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.513600    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.513759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.513619    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.513759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.513632    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.513759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.513645    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132771fd-242f-43c0-9c51-32dc14c71f98-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.514904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.514448    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-certs" pod "2dd96ad7-71c6-4389-bf36-dca4475cb5e5" (UID: "2dd96ad7-71c6-4389-bf36-dca4475cb5e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.517927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.517895    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-data" pod "132771fd-242f-43c0-9c51-32dc14c71f98" (UID: "132771fd-242f-43c0-9c51-32dc14c71f98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.518138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.517959    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-credential-socket" pod "2dd96ad7-71c6-4389-bf36-dca4475cb5e5" (UID: "2dd96ad7-71c6-4389-bf36-dca4475cb5e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.518138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.518045    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-kube-api-access-55zgm" pod "132771fd-242f-43c0-9c51-32dc14c71f98" (UID: "132771fd-242f-43c0-9c51-32dc14c71f98"). InnerVolumeSpecName "kube-api-access-55zgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.518138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.518128    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-kube-api-access-hdxqf" pod "2dd96ad7-71c6-4389-bf36-dca4475cb5e5" (UID: "2dd96ad7-71c6-4389-bf36-dca4475cb5e5"). InnerVolumeSpecName "kube-api-access-hdxqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.520035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.519738    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-certs" pod "132771fd-242f-43c0-9c51-32dc14c71f98" (UID: "132771fd-242f-43c0-9c51-32dc14c71f98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.549444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.549405    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"]
Oct 30 19:43:28.570751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.567529    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-podinfo" pod "2dd96ad7-71c6-4389-bf36-dca4475cb5e5" (UID: "2dd96ad7-71c6-4389-bf36-dca4475cb5e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:28.570751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.567529    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-envoy" pod "132771fd-242f-43c0-9c51-32dc14c71f98" (UID: "132771fd-242f-43c0-9c51-32dc14c71f98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.570751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.567830    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-istio-token" pod "132771fd-242f-43c0-9c51-32dc14c71f98" (UID: "132771fd-242f-43c0-9c51-32dc14c71f98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.570751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.567861    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/132771fd-242f-43c0-9c51-32dc14c71f98-istio-podinfo" pod "132771fd-242f-43c0-9c51-32dc14c71f98" (UID: "132771fd-242f-43c0-9c51-32dc14c71f98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:28.570751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.568184    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-token" pod "2dd96ad7-71c6-4389-bf36-dca4475cb5e5" (UID: "2dd96ad7-71c6-4389-bf36-dca4475cb5e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.570751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.570125    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-envoy" pod "2dd96ad7-71c6-4389-bf36-dca4475cb5e5" (UID: "2dd96ad7-71c6-4389-bf36-dca4475cb5e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.593810 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.593718    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"]
Oct 30 19:43:28.610938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.610896    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"]
Oct 30 19:43:28.611144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.611070    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.615007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.614621    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-data\") pod \"79ba1746-3bcc-4b27-b840-5515b4ba2950\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") "
Oct 30 19:43:28.615007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.614679    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-credential-socket\") pod \"79ba1746-3bcc-4b27-b840-5515b4ba2950\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") "
Oct 30 19:43:28.615318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.615122    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-credential-socket" pod "79ba1746-3bcc-4b27-b840-5515b4ba2950" (UID: "79ba1746-3bcc-4b27-b840-5515b4ba2950"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.615318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.615224    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-kube-api-access-qcwxr\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-kube-api-access-qcwxr\") pod \"79ba1746-3bcc-4b27-b840-5515b4ba2950\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") "
Oct 30 19:43:28.615318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.615266    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ba1746-3bcc-4b27-b840-5515b4ba2950-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ba1746-3bcc-4b27-b840-5515b4ba2950-istiod-ca-cert\") pod \"79ba1746-3bcc-4b27-b840-5515b4ba2950\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") "
Oct 30 19:43:28.615536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.615350    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-token\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-token\") pod \"79ba1746-3bcc-4b27-b840-5515b4ba2950\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") "
Oct 30 19:43:28.615536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.615451    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-socket\") pod \"79ba1746-3bcc-4b27-b840-5515b4ba2950\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") "
Oct 30 19:43:28.615536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.615490    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-podinfo\") pod \"79ba1746-3bcc-4b27-b840-5515b4ba2950\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") "
Oct 30 19:43:28.615698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.615565    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-certs\") pod \"79ba1746-3bcc-4b27-b840-5515b4ba2950\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") "
Oct 30 19:43:28.615778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.615762    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-envoy\") pod \"79ba1746-3bcc-4b27-b840-5515b4ba2950\" (UID: \"79ba1746-3bcc-4b27-b840-5515b4ba2950\") "
Oct 30 19:43:28.616105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616039    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.616105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616098    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.616268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616162    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jft\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-kube-api-access-m2jft\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.616268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616197    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e94e090-5bce-47cf-9c75-a776e08b3265-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.616374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616307    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.616426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616368    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.616484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616467    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.616553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616536    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.616649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616636    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.616727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616713    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.616794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616761    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.616860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616846    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.616920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616885    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.616920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616900    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-certs" pod "79ba1746-3bcc-4b27-b840-5515b4ba2950" (UID: "79ba1746-3bcc-4b27-b840-5515b4ba2950"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.617023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.616925    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.617142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617123    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.617201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617159    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.617201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617163    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-socket" pod "79ba1746-3bcc-4b27-b840-5515b4ba2950" (UID: "79ba1746-3bcc-4b27-b840-5515b4ba2950"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.617315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617224    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.617315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617250    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlk68\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-kube-api-access-nlk68\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.617467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617445    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617475    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617490    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617508    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55zgm\" (UniqueName: \"kubernetes.io/projected/132771fd-242f-43c0-9c51-32dc14c71f98-kube-api-access-55zgm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617526    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617541    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617557    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617572    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617587    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617602    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617616    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132771fd-242f-43c0-9c51-32dc14c71f98-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617632    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617646    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617660    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132771fd-242f-43c0-9c51-32dc14c71f98-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.617690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617675    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxqf\" (UniqueName: \"kubernetes.io/projected/2dd96ad7-71c6-4389-bf36-dca4475cb5e5-kube-api-access-hdxqf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.618266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.617800    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ba1746-3bcc-4b27-b840-5515b4ba2950-istiod-ca-cert" pod "79ba1746-3bcc-4b27-b840-5515b4ba2950" (UID: "79ba1746-3bcc-4b27-b840-5515b4ba2950"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:28.618331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.618312    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.619856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.619395    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e94e090-5bce-47cf-9c75-a776e08b3265-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.620351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.620323    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-data" pod "79ba1746-3bcc-4b27-b840-5515b4ba2950" (UID: "79ba1746-3bcc-4b27-b840-5515b4ba2950"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.620587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.620565    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.620966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.620936    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.621149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.621125    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.622916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.622895    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.624448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.624422    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-envoy" pod "79ba1746-3bcc-4b27-b840-5515b4ba2950" (UID: "79ba1746-3bcc-4b27-b840-5515b4ba2950"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.624926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.624717    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-token" pod "79ba1746-3bcc-4b27-b840-5515b4ba2950" (UID: "79ba1746-3bcc-4b27-b840-5515b4ba2950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.624926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.624795    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-kube-api-access-qcwxr" pod "79ba1746-3bcc-4b27-b840-5515b4ba2950" (UID: "79ba1746-3bcc-4b27-b840-5515b4ba2950"). InnerVolumeSpecName "kube-api-access-qcwxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.624926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.624851    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-podinfo" pod "79ba1746-3bcc-4b27-b840-5515b4ba2950" (UID: "79ba1746-3bcc-4b27-b840-5515b4ba2950"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:28.626593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.626565    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.631843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.631811    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jft\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-kube-api-access-m2jft\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.634361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.634334    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.719822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.719783    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.719822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.719835    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.720193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.719907    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.719975    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720009    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlk68\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-kube-api-access-nlk68\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720122    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720196    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720224    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720311    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720322    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsj75\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-kube-api-access-lsj75\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720409    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c66346-303b-4d1a-b566-126fa4ea48e3-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720530    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720567    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720603    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720631    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720786    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720795    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720811    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720881    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720897    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.720968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720952    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.721673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.720995    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.721673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.721265    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.721673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.721285    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.721673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.721295    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.721673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.721306    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcwxr\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-kube-api-access-qcwxr\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.721673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.721320    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ba1746-3bcc-4b27-b840-5515b4ba2950-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.721673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.721329    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ba1746-3bcc-4b27-b840-5515b4ba2950-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.721899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.721714    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.722978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.722945    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.723288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.723270    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.733747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.733680    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.733901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.733849    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlk68\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-kube-api-access-nlk68\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.791936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.789752    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:28.822533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.822500    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.822742 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.822620    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.822889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.822803    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.822889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.822837    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.822889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.822878    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsj75\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-kube-api-access-lsj75\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.823097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.822920    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.823097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.822941    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c66346-303b-4d1a-b566-126fa4ea48e3-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.823097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.823079    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.823285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.823144    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.823285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.823187    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.823528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.823496    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.823841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.823813    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.824493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.824462    6777 generic.go:356] "Generic (PLEG): container finished" podID="79ba1746-3bcc-4b27-b840-5515b4ba2950" containerID="3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7" exitCode=0
Oct 30 19:43:28.824608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.824556    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" event={"ID":"79ba1746-3bcc-4b27-b840-5515b4ba2950","Type":"ContainerDied","Data":"3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7"}
Oct 30 19:43:28.824678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.824606    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt" event={"ID":"79ba1746-3bcc-4b27-b840-5515b4ba2950","Type":"ContainerDied","Data":"ad6d754672480671fb6bd32b9f57236b3969ae543483ddc35b5ddfe381a7537b"}
Oct 30 19:43:28.824678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.824637    6777 scope.go:122] "RemoveContainer" containerID="3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7"
Oct 30 19:43:28.824832 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.824814    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"
Oct 30 19:43:28.824905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.824879    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c66346-303b-4d1a-b566-126fa4ea48e3-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.826547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.826441    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.826724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.826680    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.828169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.827953    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.829279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.829030    6777 generic.go:356] "Generic (PLEG): container finished" podID="132771fd-242f-43c0-9c51-32dc14c71f98" containerID="392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d" exitCode=0
Oct 30 19:43:28.829279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.829226    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" event={"ID":"132771fd-242f-43c0-9c51-32dc14c71f98","Type":"ContainerDied","Data":"392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d"}
Oct 30 19:43:28.829460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.829285    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v" event={"ID":"132771fd-242f-43c0-9c51-32dc14c71f98","Type":"ContainerDied","Data":"618fd6a880c63cdee8b7dafb8fbd14283af390a0c61df5a46262036555795f49"}
Oct 30 19:43:28.829460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.829236    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"
Oct 30 19:43:28.832359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.832327    6777 generic.go:356] "Generic (PLEG): container finished" podID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" containerID="a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee" exitCode=0
Oct 30 19:43:28.832591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.832571    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" event={"ID":"2dd96ad7-71c6-4389-bf36-dca4475cb5e5","Type":"ContainerDied","Data":"a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee"}
Oct 30 19:43:28.832656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.832625    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn" event={"ID":"2dd96ad7-71c6-4389-bf36-dca4475cb5e5","Type":"ContainerDied","Data":"ffa05dcb1189fc68be02facd950f56a7c80cc252bc8ccfd10342be2f6ea4f5a3"}
Oct 30 19:43:28.832741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.832718    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"
Oct 30 19:43:28.836999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.836966    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.839195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.839167    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsj75\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-kube-api-access-lsj75\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.845600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.845573    6777 scope.go:122] "RemoveContainer" containerID="3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7"
Oct 30 19:43:28.846206 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:28.846173    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7\": container with ID starting with 3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7 not found: ID does not exist" containerID="3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7"
Oct 30 19:43:28.846302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.846220    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7"} err="failed to get container status \"3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7\": rpc error: code = NotFound desc = could not find container \"3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7\": container with ID starting with 3594f74b0b02f3117f07efcbf3609ea6a9e8fb663eb87adc109b7fc601e644e7 not found: ID does not exist"
Oct 30 19:43:28.846302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.846247    6777 scope.go:122] "RemoveContainer" containerID="392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d"
Oct 30 19:43:28.862109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.861820    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:28.880423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.880386    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"]
Oct 30 19:43:28.882892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.882751    6777 scope.go:122] "RemoveContainer" containerID="a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f"
Oct 30 19:43:28.888873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.888806    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-j48lt"]
Oct 30 19:43:28.901797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.901753    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"]
Oct 30 19:43:28.912503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.912470    6777 scope.go:122] "RemoveContainer" containerID="392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d"
Oct 30 19:43:28.912972 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:28.912944    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d\": container with ID starting with 392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d not found: ID does not exist" containerID="392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d"
Oct 30 19:43:28.913027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.912987    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d"} err="failed to get container status \"392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d\": rpc error: code = NotFound desc = could not find container \"392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d\": container with ID starting with 392c305769e1acc8d5bffc5bc41c38d7c8b7bc9e648f293872e8df59c676264d not found: ID does not exist"
Oct 30 19:43:28.913027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.913014    6777 scope.go:122] "RemoveContainer" containerID="a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f"
Oct 30 19:43:28.913605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.913580    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-zxbbn"]
Oct 30 19:43:28.913762 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:28.913739    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f\": container with ID starting with a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f not found: ID does not exist" containerID="a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f"
Oct 30 19:43:28.913818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.913774    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f"} err="failed to get container status \"a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f\": rpc error: code = NotFound desc = could not find container \"a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f\": container with ID starting with a506344100d7bf5c3e3f00381cf39dfa7f2bf394455f3d92ea1cd97965975c9f not found: ID does not exist"
Oct 30 19:43:28.913818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.913798    6777 scope.go:122] "RemoveContainer" containerID="a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee"
Oct 30 19:43:28.936149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.928034    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"]
Oct 30 19:43:28.936149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.934700    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-mpf6v"]
Oct 30 19:43:28.943209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.941215    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:28.951519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.951390    6777 scope.go:122] "RemoveContainer" containerID="a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee"
Oct 30 19:43:28.952476 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:28.951988    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee\": container with ID starting with a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee not found: ID does not exist" containerID="a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee"
Oct 30 19:43:28.952476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:28.952027    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee"} err="failed to get container status \"a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee\": rpc error: code = NotFound desc = could not find container \"a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee\": container with ID starting with a089d0e36e37322eca82fecb5c795d0bad10e8584cf6d3a6be25912bdcd40aee not found: ID does not exist"
Oct 30 19:43:29.162998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.162942    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"]
Oct 30 19:43:29.165174 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:43:29.165131    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e94e090_5bce_47cf_9c75_a776e08b3265.slice/crio-3238462c4e6015183c2b4ccd8ab8930c68bc170ebe4b9dcd1b14bfd8b520b4cd WatchSource:0}: Error finding container 3238462c4e6015183c2b4ccd8ab8930c68bc170ebe4b9dcd1b14bfd8b520b4cd: Status 404 returned error can't find the container with id 3238462c4e6015183c2b4ccd8ab8930c68bc170ebe4b9dcd1b14bfd8b520b4cd
Oct 30 19:43:29.169466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.169417    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:29.169617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.169526    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:29.169617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.169573    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:29.215278 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:43:29.215206    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aadc4f7_bda3_4d70_9f64_7aab41fe0f7a.slice/crio-cd21eca7938cf6d705531f86ca962c51e21d5d22c27517db02b0dd48135f7d3f WatchSource:0}: Error finding container cd21eca7938cf6d705531f86ca962c51e21d5d22c27517db02b0dd48135f7d3f: Status 404 returned error can't find the container with id cd21eca7938cf6d705531f86ca962c51e21d5d22c27517db02b0dd48135f7d3f
Oct 30 19:43:29.217846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.217709    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"]
Oct 30 19:43:29.219590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.219547    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:29.219727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.219642    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:29.219727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.219680    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:29.272223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.272193    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"]
Oct 30 19:43:29.279839 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:43:29.279809    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c66346_303b_4d1a_b566_126fa4ea48e3.slice/crio-421f43a6b364418d62526b2613460e362cf07f36fade9c92c235423710e10ea7 WatchSource:0}: Error finding container 421f43a6b364418d62526b2613460e362cf07f36fade9c92c235423710e10ea7: Status 404 returned error can't find the container with id 421f43a6b364418d62526b2613460e362cf07f36fade9c92c235423710e10ea7
Oct 30 19:43:29.283969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.283936    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:29.284079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.284018    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:29.284079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.284048    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:29.770116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.770001    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"]
Oct 30 19:43:29.851122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.848960    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" event={"ID":"7e94e090-5bce-47cf-9c75-a776e08b3265","Type":"ContainerStarted","Data":"95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678"}
Oct 30 19:43:29.851122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.849034    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" event={"ID":"7e94e090-5bce-47cf-9c75-a776e08b3265","Type":"ContainerStarted","Data":"3238462c4e6015183c2b4ccd8ab8930c68bc170ebe4b9dcd1b14bfd8b520b4cd"}
Oct 30 19:43:29.863232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.862796    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" event={"ID":"a4c66346-303b-4d1a-b566-126fa4ea48e3","Type":"ContainerStarted","Data":"e6958c58414ce6b7c6cb5a040b603f4ca1460a747e28e8e8d72b821afb6dad45"}
Oct 30 19:43:29.863232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.862863    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" event={"ID":"a4c66346-303b-4d1a-b566-126fa4ea48e3","Type":"ContainerStarted","Data":"421f43a6b364418d62526b2613460e362cf07f36fade9c92c235423710e10ea7"}
Oct 30 19:43:29.870223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.869585    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" event={"ID":"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a","Type":"ContainerStarted","Data":"b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169"}
Oct 30 19:43:29.870223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.869659    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" event={"ID":"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a","Type":"ContainerStarted","Data":"cd21eca7938cf6d705531f86ca962c51e21d5d22c27517db02b0dd48135f7d3f"}
Oct 30 19:43:29.892189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.892118    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podStartSLOduration=151.892095065 podStartE2EDuration="2m31.892095065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:29.880640524 +0000 UTC m=+2191.877555908" watchObservedRunningTime="2026-10-30 19:43:29.892095065 +0000 UTC m=+2191.889010451"
Oct 30 19:43:29.941727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.941669    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podStartSLOduration=284.941652667 podStartE2EDuration="4m44.941652667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:29.913416942 +0000 UTC m=+2191.910332327" watchObservedRunningTime="2026-10-30 19:43:29.941652667 +0000 UTC m=+2191.938568084"
Oct 30 19:43:29.941966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.941880    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:29.944019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:29.943967    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podStartSLOduration=282.943951344 podStartE2EDuration="4m42.943951344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:29.939121975 +0000 UTC m=+2191.936037360" watchObservedRunningTime="2026-10-30 19:43:29.943951344 +0000 UTC m=+2191.940866729"
Oct 30 19:43:30.118562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:30.118526    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="132771fd-242f-43c0-9c51-32dc14c71f98" path="/var/lib/kubelet/pods/132771fd-242f-43c0-9c51-32dc14c71f98/volumes"
Oct 30 19:43:30.119175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:30.119158    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd96ad7-71c6-4389-bf36-dca4475cb5e5" path="/var/lib/kubelet/pods/2dd96ad7-71c6-4389-bf36-dca4475cb5e5/volumes"
Oct 30 19:43:30.119673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:30.119659    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ba1746-3bcc-4b27-b840-5515b4ba2950" path="/var/lib/kubelet/pods/79ba1746-3bcc-4b27-b840-5515b4ba2950/volumes"
Oct 30 19:43:30.827085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:30.794511    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:30.827085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:30.823807    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:43:30.827085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:30.823865    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 19:43:30.870220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:30.870180    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:30.876814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:30.874253    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:43:30.877162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:30.877116    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 19:43:31.042023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:31.036926    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 19:43:31.042023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:31.036991    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 19:43:31.809091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:31.805412    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:43:31.809091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:31.805485    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 19:43:31.873084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:31.872227    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:43:31.873084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:31.872298    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 19:43:31.942053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:31.941928    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 19:43:31.942053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:31.941986    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 19:43:32.829134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:32.827830    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:43:32.829134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:32.827899    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 19:43:32.867776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:32.867734    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:43:32.868137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:32.868105    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 19:43:32.977391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:32.966236    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 19:43:32.991296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:32.991248    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 19:43:33.835087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:33.803620    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:43:33.842575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:33.842516    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 19:43:33.891084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:33.873489    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:43:33.891084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:33.873557    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 19:43:33.986091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:33.948415    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 19:43:33.986091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:33.948486    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 19:43:34.295069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:34.286304    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"]
Oct 30 19:43:34.425667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:34.423560    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a2beca-sql6f"]
Oct 30 19:43:34.809087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:34.793659    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:43:34.809087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:34.793749    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 19:43:34.829153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:34.814507    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9" podUID="4feba2ef-8e08-4c91-87f5-b3c3ea3115d9" containerName="payload-processing" containerID="cri-o://7170abad143c00e586e0d806e62998f84d7f186eb050e04c56ba10121eb6764c" gracePeriod=25
Oct 30 19:43:34.878099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:34.875160    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:43:34.878099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:34.875232    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 19:43:34.943732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:34.943260    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 19:43:34.943732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:34.943319    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 19:43:35.665480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.665434    6777 generic.go:356] "Generic (PLEG): container finished" podID="4feba2ef-8e08-4c91-87f5-b3c3ea3115d9" containerID="7170abad143c00e586e0d806e62998f84d7f186eb050e04c56ba10121eb6764c" exitCode=0
Oct 30 19:43:35.666200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.666168    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9" event={"ID":"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9","Type":"ContainerDied","Data":"7170abad143c00e586e0d806e62998f84d7f186eb050e04c56ba10121eb6764c"}
Oct 30 19:43:35.697086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.671123    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:43:35.713388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.713345    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-plugins-config-volume\") pod \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\" (UID: \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\") "
Oct 30 19:43:35.713814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.713789    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-kube-api-access-89gjg\" (UniqueName: \"kubernetes.io/projected/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-kube-api-access-89gjg\") pod \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\" (UID: \"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9\") "
Oct 30 19:43:35.721546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.721504    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-plugins-config-volume" pod "4feba2ef-8e08-4c91-87f5-b3c3ea3115d9" (UID: "4feba2ef-8e08-4c91-87f5-b3c3ea3115d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:35.933454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.928540    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:35.933454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.928668    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:43:35.933454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.928721    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 19:43:35.933454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.930290    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:43:35.979080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.936138    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 19:43:35.979080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.959329    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 19:43:35.979080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.959408    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 19:43:35.987188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:35.987145    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-kube-api-access-89gjg" pod "4feba2ef-8e08-4c91-87f5-b3c3ea3115d9" (UID: "4feba2ef-8e08-4c91-87f5-b3c3ea3115d9"). InnerVolumeSpecName "kube-api-access-89gjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:36.043106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:36.036274    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89gjg\" (UniqueName: \"kubernetes.io/projected/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9-kube-api-access-89gjg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:36.139515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:36.139475    6777 scope.go:122] "RemoveContainer" containerID="ebeacfe26bb0e4c368232d869be6cbaa44b98e41971a65536fd695d083637a59"
Oct 30 19:43:36.997082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:36.969039    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:43:36.999238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:36.999160    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 19:43:37.068082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.063258    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:43:37.108085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.075250    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 19:43:37.184498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.098537    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aad22ba-304a-4f81-bec1-66fd872a8174" path="/var/lib/kubelet/pods/6aad22ba-304a-4f81-bec1-66fd872a8174/volumes"
Oct 30 19:43:37.184927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.067156    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 19:43:37.226084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.202257    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 19:43:37.474090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.472265    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:37.506093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.504193    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:43:37.548143 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:37.529693    6777 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.404s"
Oct 30 19:43:37.548143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.529753    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"]
Oct 30 19:43:37.548143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.534023    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:43:37.548143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.534147    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:43:37.746720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.626611    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.234:8000/ready\": context deadline exceeded"
Oct 30 19:43:37.746720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.666124    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:37.943122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.936161    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:43:37.943122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.936245    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 19:43:37.952262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.952195    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:38.018124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:37.994170    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:38.057086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.047998    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:43:38.076209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.076157    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 19:43:38.076497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.049223    6777 cpu_manager.go:414] "containerMap: removing container" podUID="4feba2ef-8e08-4c91-87f5-b3c3ea3115d9" containerName="payload-processing"
Oct 30 19:43:38.076632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.076612    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4feba2ef-8e08-4c91-87f5-b3c3ea3115d9" containerName="payload-processing"
Oct 30 19:43:38.087368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.077130    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="4feba2ef-8e08-4c91-87f5-b3c3ea3115d9" containerName="payload-processing"
Oct 30 19:43:38.103284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.083745    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 19:43:38.108496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.107343    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 19:43:38.132626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.128802    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"
Oct 30 19:43:38.140511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.139465    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.150724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.149324    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.150724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.149386    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.150724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.149557    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.150724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.149612    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.150724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.149756    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.150724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.149869    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.150724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.149979    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.150724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.150116    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.150724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.150194    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbpjm\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-kube-api-access-gbpjm\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.175868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.175831    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"]
Oct 30 19:43:38.176106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.175889    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9" event={"ID":"4feba2ef-8e08-4c91-87f5-b3c3ea3115d9","Type":"ContainerDied","Data":"698a9f7145536f9a88a79bcd8e18777b996f5f3b5ee75823dd9ba85619574ca0"}
Oct 30 19:43:38.176106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.175925    6777 scope.go:122] "RemoveContainer" containerID="7170abad143c00e586e0d806e62998f84d7f186eb050e04c56ba10121eb6764c"
Oct 30 19:43:38.224174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.224138    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"]
Oct 30 19:43:38.228017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.227969    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-85599ff484-jnxb9"]
Oct 30 19:43:38.252718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.251720    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.252718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.251855    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbpjm\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-kube-api-access-gbpjm\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.252718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.251947    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.252718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.251986    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.252718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.252148    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.252718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.252201    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.252718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.252417    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.252718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.252530    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.252718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.252670    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.263261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.259291    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.263261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.259661    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.263261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.260881    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.273404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.272270    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.273404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.273311    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.303953 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:38.296234    6777 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4feba2ef_8e08_4c91_87f5_b3c3ea3115d9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4feba2ef_8e08_4c91_87f5_b3c3ea3115d9.slice/crio-698a9f7145536f9a88a79bcd8e18777b996f5f3b5ee75823dd9ba85619574ca0\": RecentStats: unable to find data in memory cache]"
Oct 30 19:43:38.352396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.336828    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbpjm\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-kube-api-access-gbpjm\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.352396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.337095    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.352396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.337795    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.352396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.344139    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.510215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.509673    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:38.802308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.802164    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:38.804035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.804009    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:38.804807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.804783    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:43:38.870266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.870234    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:38.870982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.870945    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:38.872541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.872518    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:43:38.883011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.882894    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"]
Oct 30 19:43:38.887828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.887792    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:38.888011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.887884    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:38.888011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.887927    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:38.965490 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.965450    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:38.966710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.966683    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:38.967989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:38.967967    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:43:39.140906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:39.140872    6777 generic.go:356] "Generic (PLEG): container finished" podID="68c87bba-3998-433f-b361-b232946ace9e" containerID="fb23294e796cb44f42d9054a9c8527cd6ada56405f1e23bb9e7c680150eca4dc" exitCode=0
Oct 30 19:43:39.141407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:39.140942    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" event={"ID":"68c87bba-3998-433f-b361-b232946ace9e","Type":"ContainerDied","Data":"fb23294e796cb44f42d9054a9c8527cd6ada56405f1e23bb9e7c680150eca4dc"}
Oct 30 19:43:39.141407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:39.141008    6777 scope.go:122] "RemoveContainer" containerID="ebeacfe26bb0e4c368232d869be6cbaa44b98e41971a65536fd695d083637a59"
Oct 30 19:43:39.144034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:39.143992    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" event={"ID":"77cabf0b-0e2d-481f-8514-1d70954d2ae9","Type":"ContainerStarted","Data":"5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e"}
Oct 30 19:43:39.144249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:39.144123    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" event={"ID":"77cabf0b-0e2d-481f-8514-1d70954d2ae9","Type":"ContainerStarted","Data":"50192b10789a4e3c7eaa79aa36b2a2cad1209a0ebab0c086f60d4af75b2805c6"}
Oct 30 19:43:39.223243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:39.206880    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" podStartSLOduration=299.206855957 podStartE2EDuration="4m59.206855957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:39.198439463 +0000 UTC m=+2201.195354870" watchObservedRunningTime="2026-10-30 19:43:39.206855957 +0000 UTC m=+2201.203771343"
Oct 30 19:43:39.519319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:39.519158    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:40.125593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:40.125552    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4feba2ef-8e08-4c91-87f5-b3c3ea3115d9" path="/var/lib/kubelet/pods/4feba2ef-8e08-4c91-87f5-b3c3ea3115d9/volumes"
Oct 30 19:43:40.212385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:40.212351    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"]
Oct 30 19:43:40.512607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:40.512475    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:43:40.512817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:40.512630    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 19:43:41.185097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.184534    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:41.435950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.388395    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c87bba-3998-433f-b361-b232946ace9e-kube-api-access-xlcl2\" (UniqueName: \"kubernetes.io/projected/68c87bba-3998-433f-b361-b232946ace9e-kube-api-access-xlcl2\") pod \"68c87bba-3998-433f-b361-b232946ace9e\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") "
Oct 30 19:43:41.435950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.388485    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c87bba-3998-433f-b361-b232946ace9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c87bba-3998-433f-b361-b232946ace9e-maas-api-openshift-ca\") pod \"68c87bba-3998-433f-b361-b232946ace9e\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") "
Oct 30 19:43:41.435950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.388562    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68c87bba-3998-433f-b361-b232946ace9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c87bba-3998-433f-b361-b232946ace9e-maas-api-xks-ca\") pod \"68c87bba-3998-433f-b361-b232946ace9e\" (UID: \"68c87bba-3998-433f-b361-b232946ace9e\") "
Oct 30 19:43:41.435950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.396224    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c87bba-3998-433f-b361-b232946ace9e-maas-api-openshift-ca" pod "68c87bba-3998-433f-b361-b232946ace9e" (UID: "68c87bba-3998-433f-b361-b232946ace9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:41.435950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.416882    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c87bba-3998-433f-b361-b232946ace9e-kube-api-access-xlcl2" pod "68c87bba-3998-433f-b361-b232946ace9e" (UID: "68c87bba-3998-433f-b361-b232946ace9e"). InnerVolumeSpecName "kube-api-access-xlcl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:41.435950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.420913    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c87bba-3998-433f-b361-b232946ace9e-maas-api-xks-ca" pod "68c87bba-3998-433f-b361-b232946ace9e" (UID: "68c87bba-3998-433f-b361-b232946ace9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:41.495678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.495638    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlcl2\" (UniqueName: \"kubernetes.io/projected/68c87bba-3998-433f-b361-b232946ace9e-kube-api-access-xlcl2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.495940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.495919    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c87bba-3998-433f-b361-b232946ace9e-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.496050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.496037    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c87bba-3998-433f-b361-b232946ace9e-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.498217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.498189    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"
Oct 30 19:43:41.503506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.503380    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf" event={"ID":"68c87bba-3998-433f-b361-b232946ace9e","Type":"ContainerDied","Data":"e7cafe83b125ca12cfca97c9dff4af50b439f3a0ea345c605679d2e01a30f884"}
Oct 30 19:43:41.503672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.503549    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7cafe83b125ca12cfca97c9dff4af50b439f3a0ea345c605679d2e01a30f884"
Oct 30 19:43:41.511968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.511943    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:43:41.512147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.512007    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 19:43:41.605566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.605521    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"]
Oct 30 19:43:41.608236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.608208    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-2zhzf"]
Oct 30 19:43:41.806847 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:41.804981    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:43:42.203848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:42.203743    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c87bba-3998-433f-b361-b232946ace9e" path="/var/lib/kubelet/pods/68c87bba-3998-433f-b361-b232946ace9e/volumes"
Oct 30 19:43:42.511000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:42.510609    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:43:42.511000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:42.510692    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 19:43:43.370086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:43.362154    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"]
Oct 30 19:43:43.500272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:43.500215    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1558337b-rn75n"]
Oct 30 19:43:43.517509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:43.517466    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:43:43.518047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:43.518016    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 19:43:43.908516 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:43.908449    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:43:44.116185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:44.116146    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a8dda2-0d82-4069-b850-960a9f30d094" path="/var/lib/kubelet/pods/43a8dda2-0d82-4069-b850-960a9f30d094/volumes"
Oct 30 19:43:44.528261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:44.528130    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:43:44.528261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:44.528219    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 19:43:45.510603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:45.510558    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:43:45.510843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:45.510645    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 19:43:46.528087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:46.515466    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:43:46.528087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:46.515549    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 19:43:47.744697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:47.744654    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:47.745320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:47.745213    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:47.746217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:47.746191    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:50.126232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.126191    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"]
Oct 30 19:43:50.127003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.126277    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"]
Oct 30 19:43:50.127003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.126512    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy" containerID="cri-o://5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e" gracePeriod=30
Oct 30 19:43:50.127003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.126556    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy" containerID="cri-o://371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04" gracePeriod=30
Oct 30 19:43:50.130824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.130795    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"]
Oct 30 19:43:50.131052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.131006    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy" containerID="cri-o://5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83" gracePeriod=30
Oct 30 19:43:50.773115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.773069    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"]
Oct 30 19:43:50.774616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.774373    6777 cpu_manager.go:414] "containerMap: removing container" podUID="68c87bba-3998-433f-b361-b232946ace9e" containerName="revoke-keys"
Oct 30 19:43:50.774616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.774402    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c87bba-3998-433f-b361-b232946ace9e" containerName="revoke-keys"
Oct 30 19:43:50.775267 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.775246    6777 cpu_manager.go:414] "containerMap: removing container" podUID="68c87bba-3998-433f-b361-b232946ace9e" containerName="revoke-keys"
Oct 30 19:43:50.775267 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.775269    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c87bba-3998-433f-b361-b232946ace9e" containerName="revoke-keys"
Oct 30 19:43:50.775482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.775466    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c87bba-3998-433f-b361-b232946ace9e" containerName="revoke-keys"
Oct 30 19:43:50.775572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.775488    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c87bba-3998-433f-b361-b232946ace9e" containerName="revoke-keys"
Oct 30 19:43:50.775572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.775497    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c87bba-3998-433f-b361-b232946ace9e" containerName="revoke-keys"
Oct 30 19:43:50.825618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.825570    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"]
Oct 30 19:43:50.825820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.825625    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:50.828672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.828624    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:50.828856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.828791    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529jg\" (UniqueName: \"kubernetes.io/projected/905d89e6-38a1-4574-84b8-3a07922157ab-kube-api-access-529jg\") pod \"maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:50.829148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.829109    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:50.831911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.829926    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:50.831911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.830052    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:43:50.930878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.930832    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:50.931086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.930940    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:50.931086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.931038    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529jg\" (UniqueName: \"kubernetes.io/projected/905d89e6-38a1-4574-84b8-3a07922157ab-kube-api-access-529jg\") pod \"maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:50.931474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.931450    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:50.945455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.945421    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-529jg\" (UniqueName: \"kubernetes.io/projected/905d89e6-38a1-4574-84b8-3a07922157ab-kube-api-access-529jg\") pod \"maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:50.945646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:50.945560    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:51.195572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:51.195534    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:51.772749 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:43:51.772694    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905d89e6_38a1_4574_84b8_3a07922157ab.slice/crio-4190de1badcb67345ffb1d27b5e39ff634eb28a7b6e8ba0dbd6d6d4ea5808127 WatchSource:0}: Error finding container 4190de1badcb67345ffb1d27b5e39ff634eb28a7b6e8ba0dbd6d6d4ea5808127: Status 404 returned error can't find the container with id 4190de1badcb67345ffb1d27b5e39ff634eb28a7b6e8ba0dbd6d6d4ea5808127
Oct 30 19:43:51.773239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:51.773211    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"]
Oct 30 19:43:51.887619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:51.887585    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr" event={"ID":"905d89e6-38a1-4574-84b8-3a07922157ab","Type":"ContainerStarted","Data":"4190de1badcb67345ffb1d27b5e39ff634eb28a7b6e8ba0dbd6d6d4ea5808127"}
Oct 30 19:43:52.031259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:52.031221    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"]
Oct 30 19:43:52.045811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:52.045760    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ba8590-6tq5x"]
Oct 30 19:43:52.118423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:52.118378    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3450d757-8cf5-425e-adb3-4af99cca13dc" path="/var/lib/kubelet/pods/3450d757-8cf5-425e-adb3-4af99cca13dc/volumes"
Oct 30 19:43:52.899769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:52.899733    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr" event={"ID":"905d89e6-38a1-4574-84b8-3a07922157ab","Type":"ContainerStarted","Data":"c79d6da5dbae0a246998bff7a80ddf23b00e74a800a19f11339295f7e6bb63ab"}
Oct 30 19:43:52.923506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:52.923443    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr" podStartSLOduration=2.923428184 podStartE2EDuration="2.923428184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:52.920874684 +0000 UTC m=+2214.917790069" watchObservedRunningTime="2026-10-30 19:43:52.923428184 +0000 UTC m=+2214.920343635"
Oct 30 19:43:53.943513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:53.943472    6777 generic.go:356] "Generic (PLEG): container finished" podID="905d89e6-38a1-4574-84b8-3a07922157ab" containerID="c79d6da5dbae0a246998bff7a80ddf23b00e74a800a19f11339295f7e6bb63ab" exitCode=0
Oct 30 19:43:53.944224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:53.943538    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr" event={"ID":"905d89e6-38a1-4574-84b8-3a07922157ab","Type":"ContainerDied","Data":"c79d6da5dbae0a246998bff7a80ddf23b00e74a800a19f11339295f7e6bb63ab"}
Oct 30 19:43:55.239352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.239268    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:55.374667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.374630    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-openshift-ca\") pod \"905d89e6-38a1-4574-84b8-3a07922157ab\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") "
Oct 30 19:43:55.375035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.375015    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905d89e6-38a1-4574-84b8-3a07922157ab-kube-api-access-529jg\" (UniqueName: \"kubernetes.io/projected/905d89e6-38a1-4574-84b8-3a07922157ab-kube-api-access-529jg\") pod \"905d89e6-38a1-4574-84b8-3a07922157ab\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") "
Oct 30 19:43:55.375205 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.375132    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-openshift-ca" pod "905d89e6-38a1-4574-84b8-3a07922157ab" (UID: "905d89e6-38a1-4574-84b8-3a07922157ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:55.375289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.375270    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-xks-ca\") pod \"905d89e6-38a1-4574-84b8-3a07922157ab\" (UID: \"905d89e6-38a1-4574-84b8-3a07922157ab\") "
Oct 30 19:43:55.376042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.376015    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.387868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.387827    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905d89e6-38a1-4574-84b8-3a07922157ab-kube-api-access-529jg" pod "905d89e6-38a1-4574-84b8-3a07922157ab" (UID: "905d89e6-38a1-4574-84b8-3a07922157ab"). InnerVolumeSpecName "kube-api-access-529jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:55.388184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.388138    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-xks-ca" pod "905d89e6-38a1-4574-84b8-3a07922157ab" (UID: "905d89e6-38a1-4574-84b8-3a07922157ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:55.511185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.485947    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-529jg\" (UniqueName: \"kubernetes.io/projected/905d89e6-38a1-4574-84b8-3a07922157ab-kube-api-access-529jg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.511185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.509614    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d89e6-38a1-4574-84b8-3a07922157ab-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.639634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.639598    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"]
Oct 30 19:43:55.807990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.807956    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:43:55.823905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.823483    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-envoy\") pod \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") "
Oct 30 19:43:55.824130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.823935    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-data\") pod \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") "
Oct 30 19:43:55.824130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.824075    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-credential-socket\") pod \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") "
Oct 30 19:43:55.824534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.824374    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-data" pod "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" (UID: "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.824534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.824429    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-token\") pod \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") "
Oct 30 19:43:55.824712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.824682    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-socket\") pod \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") "
Oct 30 19:43:55.826339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.824769    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-kube-api-access-2nrj4\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-kube-api-access-2nrj4\") pod \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") "
Oct 30 19:43:55.826339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.824842    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-certs\") pod \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") "
Oct 30 19:43:55.826339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.824929    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istiod-ca-cert\") pod \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") "
Oct 30 19:43:55.826339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.824696    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-credential-socket" pod "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" (UID: "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.826339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.824984    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-podinfo\") pod \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\" (UID: \"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77\") "
Oct 30 19:43:55.826339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.826041    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.826339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.826083    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.826851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.826526    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istiod-ca-cert" pod "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" (UID: "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:55.827385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.826904    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-socket" pod "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" (UID: "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.827385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.826993    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-certs" pod "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" (UID: "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.836128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.834355    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-kube-api-access-2nrj4" pod "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" (UID: "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77"). InnerVolumeSpecName "kube-api-access-2nrj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:55.867777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.861721    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-token" pod "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" (UID: "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:55.867777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.865235    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-envoy" pod "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" (UID: "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.867777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.865673    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-podinfo" pod "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" (UID: "1f3dd082-6412-4cdd-93bd-e7da7cb9ba77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:55.872357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.872315    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"]
Oct 30 19:43:55.873817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.873784    6777 cpu_manager.go:414] "containerMap: removing container" podUID="68c87bba-3998-433f-b361-b232946ace9e" containerName="revoke-keys"
Oct 30 19:43:55.873817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.873814    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c87bba-3998-433f-b361-b232946ace9e" containerName="revoke-keys"
Oct 30 19:43:55.874017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.873839    6777 cpu_manager.go:414] "containerMap: removing container" podUID="905d89e6-38a1-4574-84b8-3a07922157ab" containerName="revoke-keys"
Oct 30 19:43:55.874017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.873844    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905d89e6-38a1-4574-84b8-3a07922157ab" containerName="revoke-keys"
Oct 30 19:43:55.874017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.873876    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy"
Oct 30 19:43:55.874017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.873881    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy"
Oct 30 19:43:55.874179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.874043    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="905d89e6-38a1-4574-84b8-3a07922157ab" containerName="revoke-keys"
Oct 30 19:43:55.874215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.874198    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerName="istio-proxy"
Oct 30 19:43:55.917138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.917088    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"]
Oct 30 19:43:55.917333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.917145    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.924436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.924397    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:55.927301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.927260    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.927449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.927337    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.927497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.927461    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.927540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.927503    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gws76\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-kube-api-access-gws76\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.927782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.927764    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.927886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.927867    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.928024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.927994    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.928158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.928041    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.928158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.928090    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:55.928307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.928289    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.928347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.928313    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nrj4\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-kube-api-access-2nrj4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.928347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.928327    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.928347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.928339    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.928464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.928352    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.928464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.928363    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.928464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.928379    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.938459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.938430    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:43:55.996049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.996013    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"]
Oct 30 19:43:55.998677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.998642    6777 generic.go:356] "Generic (PLEG): container finished" podID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" containerID="5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83" exitCode=0
Oct 30 19:43:55.998865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.998769    6777 cpu_manager.go:414] "containerMap: removing container" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy"
Oct 30 19:43:55.998865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.998787    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy"
Oct 30 19:43:55.998865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.998824    6777 cpu_manager.go:414] "containerMap: removing container" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy"
Oct 30 19:43:55.998865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.998832    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy"
Oct 30 19:43:55.999128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:55.998874    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"
Oct 30 19:43:56.001632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.000332    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerName="istio-proxy"
Oct 30 19:43:56.001632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.000367    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerName="istio-proxy"
Oct 30 19:43:56.010041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.009723    6777 generic.go:356] "Generic (PLEG): container finished" podID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" containerID="371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04" exitCode=0
Oct 30 19:43:56.010041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.009874    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"
Oct 30 19:43:56.015623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.015586    6777 generic.go:356] "Generic (PLEG): container finished" podID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" containerID="5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e" exitCode=0
Oct 30 19:43:56.015842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.015762    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"
Oct 30 19:43:56.029249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029216    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-podinfo\") pod \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") "
Oct 30 19:43:56.029437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029363    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-socket\") pod \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") "
Oct 30 19:43:56.029437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029433    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-credential-socket\") pod \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") "
Oct 30 19:43:56.029612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029472    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-kube-api-access-gbpjm\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-kube-api-access-gbpjm\") pod \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") "
Oct 30 19:43:56.029612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029528    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-certs\") pod \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") "
Oct 30 19:43:56.029612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029604    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-kube-api-access-lrqgm\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-kube-api-access-lrqgm\") pod \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") "
Oct 30 19:43:56.029911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029671    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istiod-ca-cert\") pod \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") "
Oct 30 19:43:56.029911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029732    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-socket" pod "77cabf0b-0e2d-481f-8514-1d70954d2ae9" (UID: "77cabf0b-0e2d-481f-8514-1d70954d2ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.029911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029779    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-data\") pod \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") "
Oct 30 19:43:56.029911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029857    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-data\") pod \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") "
Oct 30 19:43:56.029911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029905    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-podinfo\") pod \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") "
Oct 30 19:43:56.030321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.029942    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-credential-socket\") pod \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") "
Oct 30 19:43:56.030321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.030008    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istiod-ca-cert\") pod \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") "
Oct 30 19:43:56.030321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.030104    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-socket\") pod \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") "
Oct 30 19:43:56.030321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.030158    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-envoy\") pod \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") "
Oct 30 19:43:56.030321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.030209    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-envoy\") pod \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") "
Oct 30 19:43:56.030591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.030377    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-certs\") pod \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") "
Oct 30 19:43:56.030591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.030457    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-token\") pod \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\" (UID: \"77cabf0b-0e2d-481f-8514-1d70954d2ae9\") "
Oct 30 19:43:56.030591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.030547    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-token\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-token\") pod \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\" (UID: \"65f91cea-9443-4aa1-a1ef-9947c4508cc6\") "
Oct 30 19:43:56.031518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.030932    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.031518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.031097    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.031518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.031252    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.031518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.031281    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.031518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.031335    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.031911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.031552    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.031911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.031633    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.031911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.031734    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.031911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.031775    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gws76\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-kube-api-access-gws76\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.033378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.032598    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.034076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.033939    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.034076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.033960    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-credential-socket" pod "65f91cea-9443-4aa1-a1ef-9947c4508cc6" (UID: "65f91cea-9443-4aa1-a1ef-9947c4508cc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.035495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.034337    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.035495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.034466    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-certs" pod "65f91cea-9443-4aa1-a1ef-9947c4508cc6" (UID: "65f91cea-9443-4aa1-a1ef-9947c4508cc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.035495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.034564    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-podinfo" pod "77cabf0b-0e2d-481f-8514-1d70954d2ae9" (UID: "77cabf0b-0e2d-481f-8514-1d70954d2ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:56.035495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.035122    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-data" pod "65f91cea-9443-4aa1-a1ef-9947c4508cc6" (UID: "65f91cea-9443-4aa1-a1ef-9947c4508cc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.035495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.035173    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.035495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.035422    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istiod-ca-cert" pod "65f91cea-9443-4aa1-a1ef-9947c4508cc6" (UID: "65f91cea-9443-4aa1-a1ef-9947c4508cc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:56.035495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.035446    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.036178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.036125    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-socket" pod "65f91cea-9443-4aa1-a1ef-9947c4508cc6" (UID: "65f91cea-9443-4aa1-a1ef-9947c4508cc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.036592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.036290    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.036592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.036581    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-kube-api-access-lrqgm" pod "65f91cea-9443-4aa1-a1ef-9947c4508cc6" (UID: "65f91cea-9443-4aa1-a1ef-9947c4508cc6"). InnerVolumeSpecName "kube-api-access-lrqgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:56.036796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.036773    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-credential-socket" pod "77cabf0b-0e2d-481f-8514-1d70954d2ae9" (UID: "77cabf0b-0e2d-481f-8514-1d70954d2ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.036988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.036968    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-certs" pod "77cabf0b-0e2d-481f-8514-1d70954d2ae9" (UID: "77cabf0b-0e2d-481f-8514-1d70954d2ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.056301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.036881    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istiod-ca-cert" pod "77cabf0b-0e2d-481f-8514-1d70954d2ae9" (UID: "77cabf0b-0e2d-481f-8514-1d70954d2ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:56.056301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.037097    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"
Oct 30 19:43:56.056301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.038530    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-podinfo" pod "65f91cea-9443-4aa1-a1ef-9947c4508cc6" (UID: "65f91cea-9443-4aa1-a1ef-9947c4508cc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:56.056301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.040715    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-data" pod "77cabf0b-0e2d-481f-8514-1d70954d2ae9" (UID: "77cabf0b-0e2d-481f-8514-1d70954d2ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.056301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.042993    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.056301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.047779    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-envoy" pod "65f91cea-9443-4aa1-a1ef-9947c4508cc6" (UID: "65f91cea-9443-4aa1-a1ef-9947c4508cc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.056301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.052978    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-kube-api-access-gbpjm" pod "77cabf0b-0e2d-481f-8514-1d70954d2ae9" (UID: "77cabf0b-0e2d-481f-8514-1d70954d2ae9"). InnerVolumeSpecName "kube-api-access-gbpjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:56.056301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.053285    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.056301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.054490    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-envoy" pod "77cabf0b-0e2d-481f-8514-1d70954d2ae9" (UID: "77cabf0b-0e2d-481f-8514-1d70954d2ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:56.062706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.062636    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-token" pod "65f91cea-9443-4aa1-a1ef-9947c4508cc6" (UID: "65f91cea-9443-4aa1-a1ef-9947c4508cc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:56.063436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.063395    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-token" pod "77cabf0b-0e2d-481f-8514-1d70954d2ae9" (UID: "77cabf0b-0e2d-481f-8514-1d70954d2ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:56.064194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.064173    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.064416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.064385    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"]
Oct 30 19:43:56.064508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.064450    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"]
Oct 30 19:43:56.064592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.064565    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.076824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.076759    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gws76\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-kube-api-access-gws76\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097281    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"]
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097339    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"]
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097358    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" event={"ID":"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77","Type":"ContainerDied","Data":"5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83"}
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097382    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq" event={"ID":"1f3dd082-6412-4cdd-93bd-e7da7cb9ba77","Type":"ContainerDied","Data":"4e3cc31dceab1a9108a17ff03922a673a0cc9a1928c4b136d19398cf63faa11b"}
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097399    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" event={"ID":"65f91cea-9443-4aa1-a1ef-9947c4508cc6","Type":"ContainerDied","Data":"371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04"}
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097405    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097424    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6" event={"ID":"65f91cea-9443-4aa1-a1ef-9947c4508cc6","Type":"ContainerDied","Data":"ebd40157330895f05120177a85885b9e497286a9fa6b2add157ada84b5541543"}
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097453    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-dh9vq"]
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097481    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" event={"ID":"77cabf0b-0e2d-481f-8514-1d70954d2ae9","Type":"ContainerDied","Data":"5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e"}
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097507    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57" event={"ID":"77cabf0b-0e2d-481f-8514-1d70954d2ae9","Type":"ContainerDied","Data":"50192b10789a4e3c7eaa79aa36b2a2cad1209a0ebab0c086f60d4af75b2805c6"}
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097529    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr" event={"ID":"905d89e6-38a1-4574-84b8-3a07922157ab","Type":"ContainerDied","Data":"4190de1badcb67345ffb1d27b5e39ff634eb28a7b6e8ba0dbd6d6d4ea5808127"}
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097535    6777 scope.go:122] "RemoveContainer" containerID="5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83"
Oct 30 19:43:56.101441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.097543    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4190de1badcb67345ffb1d27b5e39ff634eb28a7b6e8ba0dbd6d6d4ea5808127"
Oct 30 19:43:56.125291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.125257    6777 scope.go:122] "RemoveContainer" containerID="5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83"
Oct 30 19:43:56.125745 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:56.125715    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83\": container with ID starting with 5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83 not found: ID does not exist" containerID="5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83"
Oct 30 19:43:56.125857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.125779    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83"} err="failed to get container status \"5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83\": rpc error: code = NotFound desc = could not find container \"5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83\": container with ID starting with 5c7a3fce4c9ca545bdb0a5b62e49b27f3d2b158e8f02ed8f8ab0c747d5040e83 not found: ID does not exist"
Oct 30 19:43:56.125857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.125804    6777 scope.go:122] "RemoveContainer" containerID="371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04"
Oct 30 19:43:56.134091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.133880    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3dd082-6412-4cdd-93bd-e7da7cb9ba77" path="/var/lib/kubelet/pods/1f3dd082-6412-4cdd-93bd-e7da7cb9ba77/volumes"
Oct 30 19:43:56.135018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.134644    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.135018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.134691    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.135018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.134751    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.135018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.134808    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.135018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.134852    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.135888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.135434    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.135888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.135647    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.135888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.135683    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.135888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.135716    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpst\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-kube-api-access-4mpst\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.135888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.135750    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-kube-api-access-td49m\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.135888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.135783    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.135888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.135810    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.135888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.135858    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3631e6-8e99-43f2-8e80-44524d73d34a-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.136383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136049    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.136383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136180    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.136383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136209    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.136383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136235    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.136383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136285    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136433    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136449    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136464    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136479    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136495    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136509    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136524    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136541    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136556    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbpjm\" (UniqueName: \"kubernetes.io/projected/77cabf0b-0e2d-481f-8514-1d70954d2ae9-kube-api-access-gbpjm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136572    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136585    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqgm\" (UniqueName: \"kubernetes.io/projected/65f91cea-9443-4aa1-a1ef-9947c4508cc6-kube-api-access-lrqgm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136599    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136613    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136629    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136642    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f91cea-9443-4aa1-a1ef-9947c4508cc6-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136656    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cabf0b-0e2d-481f-8514-1d70954d2ae9-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.136763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.136670    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cabf0b-0e2d-481f-8514-1d70954d2ae9-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:56.151177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.151137    6777 scope.go:122] "RemoveContainer" containerID="371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04"
Oct 30 19:43:56.151531 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:56.151504    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04\": container with ID starting with 371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04 not found: ID does not exist" containerID="371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04"
Oct 30 19:43:56.151613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.151544    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04"} err="failed to get container status \"371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04\": rpc error: code = NotFound desc = could not find container \"371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04\": container with ID starting with 371cf0190073c258481bea4373ec46e37de9a4b631e78e6a428d077e19cc3d04 not found: ID does not exist"
Oct 30 19:43:56.151613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.151567    6777 scope.go:122] "RemoveContainer" containerID="5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e"
Oct 30 19:43:56.167598 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.167573    6777 scope.go:122] "RemoveContainer" containerID="5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e"
Oct 30 19:43:56.167947 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:43:56.167925    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e\": container with ID starting with 5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e not found: ID does not exist" containerID="5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e"
Oct 30 19:43:56.168021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.167956    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e"} err="failed to get container status \"5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e\": rpc error: code = NotFound desc = could not find container \"5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e\": container with ID starting with 5459b83757d58b50591a2d6cb2d0fcc4a059c79f082f2367252c2baa2cf84f4e not found: ID does not exist"
Oct 30 19:43:56.237874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.237826    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.238113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.237950    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.238113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238026    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238126    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238157    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpst\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-kube-api-access-4mpst\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238193    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-kube-api-access-td49m\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238229    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238254    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238303    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3631e6-8e99-43f2-8e80-44524d73d34a-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238393    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238524    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238553    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238580    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238582    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238607    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238668    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238711    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238808    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238840    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238885    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238907    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.238977    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.239156    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.239182    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.239384    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3631e6-8e99-43f2-8e80-44524d73d34a-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.239411    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.239445    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.239679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.239466    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.241667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.241639    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.241824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.241806    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.241890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.241820    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.242241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.242217    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.251827 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.251791    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpst\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-kube-api-access-4mpst\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.252934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.252760    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-kube-api-access-td49m\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.253471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.253445    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.253842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.253622    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.253842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.253782    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:56.350117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.349999    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"]
Oct 30 19:43:56.371077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.371031    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-wgg57"]
Oct 30 19:43:56.380106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.380074    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"]
Oct 30 19:43:56.405321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.405280    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-mbxl6"]
Oct 30 19:43:56.405599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.405506    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:56.440795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.440735    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:56.686957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.686397    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"]
Oct 30 19:43:56.688810 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:43:56.688781    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbed3e02_2c98_4da9_9457_e3fdb48be2de.slice/crio-78861789841cd896d1554fedbdee4e3802ae2ecef5f0dd71c7784593a9060fd7 WatchSource:0}: Error finding container 78861789841cd896d1554fedbdee4e3802ae2ecef5f0dd71c7784593a9060fd7: Status 404 returned error can't find the container with id 78861789841cd896d1554fedbdee4e3802ae2ecef5f0dd71c7784593a9060fd7
Oct 30 19:43:56.693012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.692961    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:56.693186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.693088    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:56.693186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.693138    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:56.749952 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:43:56.749916    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bc02d6_1cd9_44a3_9cb8_97cf81560051.slice/crio-90ca5ab9ebd2bffa6f2b2b1fb2e44c5fb26525b2079faff67a349af28d867a14 WatchSource:0}: Error finding container 90ca5ab9ebd2bffa6f2b2b1fb2e44c5fb26525b2079faff67a349af28d867a14: Status 404 returned error can't find the container with id 90ca5ab9ebd2bffa6f2b2b1fb2e44c5fb26525b2079faff67a349af28d867a14
Oct 30 19:43:56.752340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.752311    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"]
Oct 30 19:43:56.754396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.754361    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:56.754528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.754438    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:56.754528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.754469    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:56.801385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.801299    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"]
Oct 30 19:43:56.801567 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:43:56.801417    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3631e6_8e99_43f2_8e80_44524d73d34a.slice/crio-117a40f24ef0f9b40251407d31c35b62b37c1a98c5318ba5a5255898b8cb56ea WatchSource:0}: Error finding container 117a40f24ef0f9b40251407d31c35b62b37c1a98c5318ba5a5255898b8cb56ea: Status 404 returned error can't find the container with id 117a40f24ef0f9b40251407d31c35b62b37c1a98c5318ba5a5255898b8cb56ea
Oct 30 19:43:56.805433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.805370    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:56.805522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.805446    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:56.805522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:56.805479    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:43:57.034421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.034381    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" event={"ID":"05bc02d6-1cd9-44a3-9cb8-97cf81560051","Type":"ContainerStarted","Data":"0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859"}
Oct 30 19:43:57.034675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.034449    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" event={"ID":"05bc02d6-1cd9-44a3-9cb8-97cf81560051","Type":"ContainerStarted","Data":"90ca5ab9ebd2bffa6f2b2b1fb2e44c5fb26525b2079faff67a349af28d867a14"}
Oct 30 19:43:57.041581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.041538    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" event={"ID":"dbed3e02-2c98-4da9-9457-e3fdb48be2de","Type":"ContainerStarted","Data":"9d580967d65f3d21e2627daff29595f31239b50b3b6058b2233164ad617aed72"}
Oct 30 19:43:57.041795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.041613    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" event={"ID":"dbed3e02-2c98-4da9-9457-e3fdb48be2de","Type":"ContainerStarted","Data":"78861789841cd896d1554fedbdee4e3802ae2ecef5f0dd71c7784593a9060fd7"}
Oct 30 19:43:57.048459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.047353    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" event={"ID":"0b3631e6-8e99-43f2-8e80-44524d73d34a","Type":"ContainerStarted","Data":"551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7"}
Oct 30 19:43:57.048459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.047417    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" event={"ID":"0b3631e6-8e99-43f2-8e80-44524d73d34a","Type":"ContainerStarted","Data":"117a40f24ef0f9b40251407d31c35b62b37c1a98c5318ba5a5255898b8cb56ea"}
Oct 30 19:43:57.094837 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.094698    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podStartSLOduration=367.094673747 podStartE2EDuration="6m7.094673747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:57.062590709 +0000 UTC m=+2219.059506095" watchObservedRunningTime="2026-10-30 19:43:57.094673747 +0000 UTC m=+2219.091589133"
Oct 30 19:43:57.096305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.096237    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podStartSLOduration=367.096221679 podStartE2EDuration="6m7.096221679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:57.089781377 +0000 UTC m=+2219.086696764" watchObservedRunningTime="2026-10-30 19:43:57.096221679 +0000 UTC m=+2219.093137065"
Oct 30 19:43:57.128645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.128574    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podStartSLOduration=357.128549734 podStartE2EDuration="5m57.128549734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:57.122153146 +0000 UTC m=+2219.119068524" watchObservedRunningTime="2026-10-30 19:43:57.128549734 +0000 UTC m=+2219.125465119"
Oct 30 19:43:57.255405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.255346    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:43:57.256522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.256494    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:43:57.256724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.256698    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 19:43:57.410139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.409529    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:43:57.412711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.412357    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:43:57.412711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.412425    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 30 19:43:57.448681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.447139    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:43:57.449367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.449019    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:43:57.449367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:57.449095    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 19:43:58.344145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:58.343389    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f91cea-9443-4aa1-a1ef-9947c4508cc6" path="/var/lib/kubelet/pods/65f91cea-9443-4aa1-a1ef-9947c4508cc6/volumes"
Oct 30 19:43:58.344647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:58.344475    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:43:58.344647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:58.344552    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 19:43:58.419084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:58.392699    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77cabf0b-0e2d-481f-8514-1d70954d2ae9" path="/var/lib/kubelet/pods/77cabf0b-0e2d-481f-8514-1d70954d2ae9/volumes"
Oct 30 19:43:58.442154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:58.442114    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:43:58.442453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:58.442419    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 19:43:58.483684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:58.473149    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:43:58.483684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:58.473209    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 30 19:43:59.298087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:59.272507    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:43:59.298427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:59.298386    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 19:43:59.540088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:59.513218    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:43:59.540088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:59.513292    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 30 19:43:59.817102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:59.793357    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:43:59.817102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:43:59.793424    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 19:44:00.467122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:00.375480    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:00.541631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:00.541585    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 19:44:00.542901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:00.412219    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:00.544087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:00.544040    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:00.545632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:00.545546    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:44:00.545632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:00.545607    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 19:44:00.545901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:00.545876    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:44:00.546023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:00.545946    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 30 19:44:00.739935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:00.739826    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"]
Oct 30 19:44:01.251080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:01.247375    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"]
Oct 30 19:44:01.280375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:01.280340    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:01.280662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:01.280631    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 19:44:01.713840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:01.635499    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:44:01.713840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:01.635561    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 30 19:44:01.713840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:01.635969    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:44:01.713840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:01.636004    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 19:44:01.871442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:01.830783    6777 scope.go:122] "RemoveContainer" containerID="a6f8c7c6783cba1587526b865245c4530665608b5cb9b4b4decf06c166bfb27a"
Oct 30 19:44:01.929264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:01.875425    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:02.295083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.267237    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:44:02.295083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.268940    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.236:8000/ready\": context deadline exceeded"
Oct 30 19:44:02.295083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.291405    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:02.295083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.291465    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 19:44:02.295083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.291710    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:44:02.295083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.291910    6777 scope.go:122] "RemoveContainer" containerID="3a62f769d7539cc8f621ea00a90de9dff6e29cfa8178b89a284fbf0c7127e2cf"
Oct 30 19:44:02.336086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.318414    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:02.369415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.369363    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:02.419280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.419240    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:44:02.422090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.419534    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 30 19:44:02.447087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.442463    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:44:02.447087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.442521    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 19:44:02.540233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.540200    6777 scope.go:122] "RemoveContainer" containerID="1c546cea20f54e0aa56a0acd5c9f14fa069d66ef595b9e044091750f36cc7a77"
Oct 30 19:44:02.728087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:02.719259    6777 scope.go:122] "RemoveContainer" containerID="adfcc77d2983a298328761504c870d3f2749517f11d83aec75301e6a0522051b"
Oct 30 19:44:03.042668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:03.042633    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"]
Oct 30 19:44:03.052377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:03.048715    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd7e582-nfc2d"]
Oct 30 19:44:03.418219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:03.403052    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:03.418219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:03.403132    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 19:44:03.418219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:03.413080    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:44:03.418219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:03.413138    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 30 19:44:03.521165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:03.494005    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:44:03.521165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:03.494086    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 19:44:03.991449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:03.979676    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"]
Oct 30 19:44:04.375184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.346570    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.236:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:04.375184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.347147    6777 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:04.375184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.349254    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:44:04.375184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.365474    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:04.375184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.365899    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:04.379212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.379170    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:44:04.385088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.381192    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:04.385088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.381251    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 19:44:04.393639 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.393355    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:44:04.423348 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.423300    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:04.446131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.425395    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28" podUID="d9cbeb26-fde8-410b-b600-b3d536fed7c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:04.468574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.436173    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:44:04.475083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.470824    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:44:04.475083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.470885    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 30 19:44:04.511304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.506207    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:44:04.511304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.506259    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 19:44:04.522117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.521419    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e85a7b4-91c1-446b-9160-1ca4d7212bfa" path="/var/lib/kubelet/pods/7e85a7b4-91c1-446b-9160-1ca4d7212bfa/volumes"
Oct 30 19:44:04.527184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.527043    6777 generic.go:356] "Generic (PLEG): container finished" podID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerID="d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce" exitCode=0
Oct 30 19:44:04.528089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.527648    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" event={"ID":"3806290b-05d3-4460-8b13-eeef5668d2e4","Type":"ContainerDied","Data":"d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce"}
Oct 30 19:44:04.530277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.529465    6777 scope.go:122] "RemoveContainer" containerID="d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce"
Oct 30 19:44:04.546138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.546013    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:44:04.546304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.546168    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:44:04.546304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:04.546247    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:44:05.269985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.269903    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:44:05.270799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.270768    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:44:05.272221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.272191    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:44:05.495123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.495038    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:44:05.496442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.496416    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:44:05.497648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.497621    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:44:05.554311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.554270    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" event={"ID":"3806290b-05d3-4460-8b13-eeef5668d2e4","Type":"ContainerStarted","Data":"b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0"}
Oct 30 19:44:05.768580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.768486    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:44:05.769156 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.769110    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:44:05.770791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:05.770770    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:44:06.029209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:06.029099    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:44:06.030142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:06.030051    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:44:06.030293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:06.030157    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 19:44:06.272083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:06.271846    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9" podUID="1299b4be-5ada-45f6-b7bb-9e0e25e6854f" containerName="payload-processing" containerID="cri-o://0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81" gracePeriod=25
Oct 30 19:44:07.029125 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:07.029086    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:44:07.029341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:07.029155    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 19:44:08.028994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:08.028952    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:44:08.029413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:08.029013    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 19:44:08.074289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:08.074181    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"]
Oct 30 19:44:08.078644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:08.078606    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c4aba99-nrdbb"]
Oct 30 19:44:08.115922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:08.115885    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a71ab85f-50a8-4ef0-9490-08579fca34bf" path="/var/lib/kubelet/pods/a71ab85f-50a8-4ef0-9490-08579fca34bf/volumes"
Oct 30 19:44:09.029478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:09.029443    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:44:09.029940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:09.029503    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 19:44:10.039968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:10.033522    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:44:10.039968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:10.033585    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 19:44:10.702101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:10.702002    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main" containerID="cri-o://a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4" gracePeriod=45
Oct 30 19:44:11.029637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.029596    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:44:11.029825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.029678    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 19:44:11.103789 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.103747    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:44:11.225308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.225268    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-kube-api-access-8g6l5\" (UniqueName: \"kubernetes.io/projected/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-kube-api-access-8g6l5\") pod \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") "
Oct 30 19:44:11.225523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.225313    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tmp-dir\") pod \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") "
Oct 30 19:44:11.225605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.225584    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-dshm\") pod \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") "
Oct 30 19:44:11.225665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.225642    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tls-certs\") pod \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") "
Oct 30 19:44:11.225665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.225645    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tmp-dir" pod "8a27f460-fdfb-49eb-a8ee-2894d9d885ad" (UID: "8a27f460-fdfb-49eb-a8ee-2894d9d885ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:11.225783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.225720    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-model-cache\") pod \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") "
Oct 30 19:44:11.225862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.225846    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-home\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-home\") pod \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\" (UID: \"8a27f460-fdfb-49eb-a8ee-2894d9d885ad\") "
Oct 30 19:44:11.226189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.226164    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-model-cache" pod "8a27f460-fdfb-49eb-a8ee-2894d9d885ad" (UID: "8a27f460-fdfb-49eb-a8ee-2894d9d885ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:11.226517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.226498    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-home" pod "8a27f460-fdfb-49eb-a8ee-2894d9d885ad" (UID: "8a27f460-fdfb-49eb-a8ee-2894d9d885ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:11.227084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.227049    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.227157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.227092    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.227157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.227109    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.228233 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.228209    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-kube-api-access-8g6l5" pod "8a27f460-fdfb-49eb-a8ee-2894d9d885ad" (UID: "8a27f460-fdfb-49eb-a8ee-2894d9d885ad"). InnerVolumeSpecName "kube-api-access-8g6l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:11.228541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.228517    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-dshm" pod "8a27f460-fdfb-49eb-a8ee-2894d9d885ad" (UID: "8a27f460-fdfb-49eb-a8ee-2894d9d885ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:11.229600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.229578    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tls-certs" pod "8a27f460-fdfb-49eb-a8ee-2894d9d885ad" (UID: "8a27f460-fdfb-49eb-a8ee-2894d9d885ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:11.328913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.328803    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g6l5\" (UniqueName: \"kubernetes.io/projected/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-kube-api-access-8g6l5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.328913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.328857    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.328913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.328867    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a27f460-fdfb-49eb-a8ee-2894d9d885ad-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.626849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.626752    6777 generic.go:356] "Generic (PLEG): container finished" podID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerID="a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4" exitCode=0
Oct 30 19:44:11.626849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.626793    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"
Oct 30 19:44:11.627124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.626841    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" event={"ID":"8a27f460-fdfb-49eb-a8ee-2894d9d885ad","Type":"ContainerDied","Data":"a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4"}
Oct 30 19:44:11.627124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.626887    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf" event={"ID":"8a27f460-fdfb-49eb-a8ee-2894d9d885ad","Type":"ContainerDied","Data":"5e601ce3222f9bbb1c3b559d3ce67502612528c945fc15a927ed2e0b2848ac2e"}
Oct 30 19:44:11.627124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.626908    6777 scope.go:122] "RemoveContainer" containerID="a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4"
Oct 30 19:44:11.716552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.716516    6777 scope.go:122] "RemoveContainer" containerID="a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4"
Oct 30 19:44:11.717624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.717597    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"]
Oct 30 19:44:11.718264 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:11.718237    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4\": container with ID starting with a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4 not found: ID does not exist" containerID="a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4"
Oct 30 19:44:11.718351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.718277    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4"} err="failed to get container status \"a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4\": rpc error: code = NotFound desc = could not find container \"a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4\": container with ID starting with a1b21367c1eccb9608748d5652ff7fd8c168eae7b5b37e41e503583c327fced4 not found: ID does not exist"
Oct 30 19:44:11.724440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:11.724412    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/simulator-w6-bce71f-kserve-69f47c5589-nbrzf"]
Oct 30 19:44:12.033906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:12.033875    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:44:12.034340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:12.034317    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:44:12.035051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:12.035025    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:44:12.117383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:12.117349    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" path="/var/lib/kubelet/pods/8a27f460-fdfb-49eb-a8ee-2894d9d885ad/volumes"
Oct 30 19:44:13.644194 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:13.644156    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:44:14.052261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.052226    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"]
Oct 30 19:44:14.057625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.057592    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7edc0d-zqfsh"]
Oct 30 19:44:14.116741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.116705    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836fdc23-6f7d-4a60-b829-d2969da31873" path="/var/lib/kubelet/pods/836fdc23-6f7d-4a60-b829-d2969da31873/volumes"
Oct 30 19:44:14.430304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.418155    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"]
Oct 30 19:44:14.430304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.421016    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main"
Oct 30 19:44:14.430304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.421040    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main"
Oct 30 19:44:14.430304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.421569    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a27f460-fdfb-49eb-a8ee-2894d9d885ad" containerName="main"
Oct 30 19:44:14.430304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.429871    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.434915 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.433140    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"]
Oct 30 19:44:14.494280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.494250    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:14.494923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.494298    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:44:14.508717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.508677    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsnwv\" (UniqueName: \"kubernetes.io/projected/30def2e4-b284-4a35-97f7-4f92fdc1920d-kube-api-access-tsnwv\") pod \"maas-api-revoke-sub-ffe14008-q5sz8\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.508895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.508721    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9" podUID="1299b4be-5ada-45f6-b7bb-9e0e25e6854f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:14.508895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.508878    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffe14008-q5sz8\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.509006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.508982    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffe14008-q5sz8\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.611416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.611378    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsnwv\" (UniqueName: \"kubernetes.io/projected/30def2e4-b284-4a35-97f7-4f92fdc1920d-kube-api-access-tsnwv\") pod \"maas-api-revoke-sub-ffe14008-q5sz8\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.611614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.611517    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffe14008-q5sz8\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.611614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.611565    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffe14008-q5sz8\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.612429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.612400    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffe14008-q5sz8\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.614133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.614090    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffe14008-q5sz8\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.635935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.635902    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsnwv\" (UniqueName: \"kubernetes.io/projected/30def2e4-b284-4a35-97f7-4f92fdc1920d-kube-api-access-tsnwv\") pod \"maas-api-revoke-sub-ffe14008-q5sz8\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:14.780253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:14.779014    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:15.100101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:15.099201    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"]
Oct 30 19:44:15.103228 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:44:15.103193    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30def2e4_b284_4a35_97f7_4f92fdc1920d.slice/crio-3a51818fc866c245174b0e45717be847771c074d077ae91b2d9cca68821445fe WatchSource:0}: Error finding container 3a51818fc866c245174b0e45717be847771c074d077ae91b2d9cca68821445fe: Status 404 returned error can't find the container with id 3a51818fc866c245174b0e45717be847771c074d077ae91b2d9cca68821445fe
Oct 30 19:44:15.670014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:15.669917    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8" event={"ID":"30def2e4-b284-4a35-97f7-4f92fdc1920d","Type":"ContainerStarted","Data":"c21652150aabb9214115cb43d1a48e0bb819a1e0d34ca3155bb14c185e61047e"}
Oct 30 19:44:15.670191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:15.670024    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8" event={"ID":"30def2e4-b284-4a35-97f7-4f92fdc1920d","Type":"ContainerStarted","Data":"3a51818fc866c245174b0e45717be847771c074d077ae91b2d9cca68821445fe"}
Oct 30 19:44:15.691596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:15.691538    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8" podStartSLOduration=1.6915179409999999 podStartE2EDuration="1.691517941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:15.689259982 +0000 UTC m=+2237.686175368" watchObservedRunningTime="2026-10-30 19:44:15.691517941 +0000 UTC m=+2237.688433327"
Oct 30 19:44:16.682716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:16.682678    6777 generic.go:356] "Generic (PLEG): container finished" podID="30def2e4-b284-4a35-97f7-4f92fdc1920d" containerID="c21652150aabb9214115cb43d1a48e0bb819a1e0d34ca3155bb14c185e61047e" exitCode=0
Oct 30 19:44:16.683146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:16.682759    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8" event={"ID":"30def2e4-b284-4a35-97f7-4f92fdc1920d","Type":"ContainerDied","Data":"c21652150aabb9214115cb43d1a48e0bb819a1e0d34ca3155bb14c185e61047e"}
Oct 30 19:44:17.347457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.347417    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerName="main" containerID="cri-o://9f56274d07464ed90c85a93c33e4196a14c932c53005f4c6488fcd6c33f4c5b4" gracePeriod=44
Oct 30 19:44:17.755169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.745982    6777 generic.go:356] "Generic (PLEG): container finished" podID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerID="9f56274d07464ed90c85a93c33e4196a14c932c53005f4c6488fcd6c33f4c5b4" exitCode=0
Oct 30 19:44:17.755169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.746445    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" event={"ID":"7dea1f43-5e0f-4c69-983a-e3c9f166d224","Type":"ContainerDied","Data":"9f56274d07464ed90c85a93c33e4196a14c932c53005f4c6488fcd6c33f4c5b4"}
Oct 30 19:44:17.783591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.783256    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.839025    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-model-cache\") pod \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") "
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.839253    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tmp-dir\") pod \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") "
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.839298    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dea1f43-5e0f-4c69-983a-e3c9f166d224-kube-api-access-f5llv\" (UniqueName: \"kubernetes.io/projected/7dea1f43-5e0f-4c69-983a-e3c9f166d224-kube-api-access-f5llv\") pod \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") "
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.839367    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-home\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-home\") pod \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") "
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.839408    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-dshm\") pod \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") "
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.839513    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-model-cache" pod "7dea1f43-5e0f-4c69-983a-e3c9f166d224" (UID: "7dea1f43-5e0f-4c69-983a-e3c9f166d224"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.839708    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tls-certs\") pod \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\" (UID: \"7dea1f43-5e0f-4c69-983a-e3c9f166d224\") "
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.840082    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tmp-dir" pod "7dea1f43-5e0f-4c69-983a-e3c9f166d224" (UID: "7dea1f43-5e0f-4c69-983a-e3c9f166d224"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.840723    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:17.841176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.840743    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:17.842339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.842023    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-home" pod "7dea1f43-5e0f-4c69-983a-e3c9f166d224" (UID: "7dea1f43-5e0f-4c69-983a-e3c9f166d224"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:17.846898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.846859    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tls-certs" pod "7dea1f43-5e0f-4c69-983a-e3c9f166d224" (UID: "7dea1f43-5e0f-4c69-983a-e3c9f166d224"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:17.857883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.857834    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dea1f43-5e0f-4c69-983a-e3c9f166d224-kube-api-access-f5llv" pod "7dea1f43-5e0f-4c69-983a-e3c9f166d224" (UID: "7dea1f43-5e0f-4c69-983a-e3c9f166d224"). InnerVolumeSpecName "kube-api-access-f5llv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:17.861038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.860998    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-dshm" pod "7dea1f43-5e0f-4c69-983a-e3c9f166d224" (UID: "7dea1f43-5e0f-4c69-983a-e3c9f166d224"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:17.943511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.943468    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5llv\" (UniqueName: \"kubernetes.io/projected/7dea1f43-5e0f-4c69-983a-e3c9f166d224-kube-api-access-f5llv\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:17.943511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.943512    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:17.943511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.943526    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dea1f43-5e0f-4c69-983a-e3c9f166d224-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:17.943511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.943537    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dea1f43-5e0f-4c69-983a-e3c9f166d224-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:17.976241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:17.976169    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:18.045016 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.044978    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-openshift-ca\") pod \"30def2e4-b284-4a35-97f7-4f92fdc1920d\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") "
Oct 30 19:44:18.045224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.045177    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-xks-ca\") pod \"30def2e4-b284-4a35-97f7-4f92fdc1920d\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") "
Oct 30 19:44:18.045300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.045283    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30def2e4-b284-4a35-97f7-4f92fdc1920d-kube-api-access-tsnwv\" (UniqueName: \"kubernetes.io/projected/30def2e4-b284-4a35-97f7-4f92fdc1920d-kube-api-access-tsnwv\") pod \"30def2e4-b284-4a35-97f7-4f92fdc1920d\" (UID: \"30def2e4-b284-4a35-97f7-4f92fdc1920d\") "
Oct 30 19:44:18.045424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.045394    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-openshift-ca" pod "30def2e4-b284-4a35-97f7-4f92fdc1920d" (UID: "30def2e4-b284-4a35-97f7-4f92fdc1920d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:18.046274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.046245    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:18.047454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.047423    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-xks-ca" pod "30def2e4-b284-4a35-97f7-4f92fdc1920d" (UID: "30def2e4-b284-4a35-97f7-4f92fdc1920d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:18.047566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.047505    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30def2e4-b284-4a35-97f7-4f92fdc1920d-kube-api-access-tsnwv" pod "30def2e4-b284-4a35-97f7-4f92fdc1920d" (UID: "30def2e4-b284-4a35-97f7-4f92fdc1920d"). InnerVolumeSpecName "kube-api-access-tsnwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:18.148885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.148853    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30def2e4-b284-4a35-97f7-4f92fdc1920d-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:18.148885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.148881    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsnwv\" (UniqueName: \"kubernetes.io/projected/30def2e4-b284-4a35-97f7-4f92fdc1920d-kube-api-access-tsnwv\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:18.592538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.592494    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.234:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:18.756423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.756342    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"
Oct 30 19:44:18.756423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.756376    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8" event={"ID":"30def2e4-b284-4a35-97f7-4f92fdc1920d","Type":"ContainerDied","Data":"3a51818fc866c245174b0e45717be847771c074d077ae91b2d9cca68821445fe"}
Oct 30 19:44:18.756938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.756429    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a51818fc866c245174b0e45717be847771c074d077ae91b2d9cca68821445fe"
Oct 30 19:44:18.758293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.758269    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"
Oct 30 19:44:18.758427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.758310    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv" event={"ID":"7dea1f43-5e0f-4c69-983a-e3c9f166d224","Type":"ContainerDied","Data":"f2823e9f2f4886551ef3e924e7a2ae86221987c32cc9c565631f0f230ab2d139"}
Oct 30 19:44:18.758427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.758365    6777 scope.go:122] "RemoveContainer" containerID="9f56274d07464ed90c85a93c33e4196a14c932c53005f4c6488fcd6c33f4c5b4"
Oct 30 19:44:18.779169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.779128    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"]
Oct 30 19:44:18.785269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:18.785235    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/rate-test-model-ddb6ea-kserve-6dd7bf6f6f-7s6vv"]
Oct 30 19:44:19.207544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.207507    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"]
Oct 30 19:44:19.208390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.208372    6777 cpu_manager.go:414] "containerMap: removing container" podUID="30def2e4-b284-4a35-97f7-4f92fdc1920d" containerName="revoke-keys"
Oct 30 19:44:19.208446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.208392    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30def2e4-b284-4a35-97f7-4f92fdc1920d" containerName="revoke-keys"
Oct 30 19:44:19.208446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.208412    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerName="main"
Oct 30 19:44:19.208446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.208418    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerName="main"
Oct 30 19:44:19.208607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.208596    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="30def2e4-b284-4a35-97f7-4f92fdc1920d" containerName="revoke-keys"
Oct 30 19:44:19.208649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.208611    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" containerName="main"
Oct 30 19:44:19.214009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.213984    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.218570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.218544    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:19.219007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.218986    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:44:19.221439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.221414    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"]
Oct 30 19:44:19.296252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.296199    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.296428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.296373    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.296529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.296513    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntc4\" (UniqueName: \"kubernetes.io/projected/441166f0-c2a7-4003-b34a-d5bc35b66b73-kube-api-access-nntc4\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.398813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.398772    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntc4\" (UniqueName: \"kubernetes.io/projected/441166f0-c2a7-4003-b34a-d5bc35b66b73-kube-api-access-nntc4\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.399014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.399000    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.399136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.399118    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.399744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.399712    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.401678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.401654    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.412019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.411979    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntc4\" (UniqueName: \"kubernetes.io/projected/441166f0-c2a7-4003-b34a-d5bc35b66b73-kube-api-access-nntc4\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.444648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.444607    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main" containerID="cri-o://ca250a79893739698f92dc3331c32438020b4b16a2e546ef3bd6c054cf177d60" gracePeriod=45
Oct 30 19:44:19.544868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.544828    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:19.774184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.774091    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" event={"ID":"ef751334-a08e-4bd0-afd8-d700e073bfff","Type":"ContainerDied","Data":"ca250a79893739698f92dc3331c32438020b4b16a2e546ef3bd6c054cf177d60"}
Oct 30 19:44:19.774184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.774098    6777 generic.go:356] "Generic (PLEG): container finished" podID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerID="ca250a79893739698f92dc3331c32438020b4b16a2e546ef3bd6c054cf177d60" exitCode=0
Oct 30 19:44:19.786973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.786945    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:44:19.837398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.837354    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"]
Oct 30 19:44:19.915737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.915702    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-model-cache\") pod \"ef751334-a08e-4bd0-afd8-d700e073bfff\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") "
Oct 30 19:44:19.915946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.915816    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef751334-a08e-4bd0-afd8-d700e073bfff-kube-api-access-7dcsl\" (UniqueName: \"kubernetes.io/projected/ef751334-a08e-4bd0-afd8-d700e073bfff-kube-api-access-7dcsl\") pod \"ef751334-a08e-4bd0-afd8-d700e073bfff\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") "
Oct 30 19:44:19.916032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916002    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-tmp-dir\") pod \"ef751334-a08e-4bd0-afd8-d700e073bfff\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") "
Oct 30 19:44:19.916149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916040    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-model-cache" pod "ef751334-a08e-4bd0-afd8-d700e073bfff" (UID: "ef751334-a08e-4bd0-afd8-d700e073bfff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.916202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916158    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef751334-a08e-4bd0-afd8-d700e073bfff-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef751334-a08e-4bd0-afd8-d700e073bfff-tls-certs\") pod \"ef751334-a08e-4bd0-afd8-d700e073bfff\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") "
Oct 30 19:44:19.916202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916182    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-dshm\") pod \"ef751334-a08e-4bd0-afd8-d700e073bfff\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") "
Oct 30 19:44:19.916386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916221    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-home\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-home\") pod \"ef751334-a08e-4bd0-afd8-d700e073bfff\" (UID: \"ef751334-a08e-4bd0-afd8-d700e073bfff\") "
Oct 30 19:44:19.916386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916305    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-tmp-dir" pod "ef751334-a08e-4bd0-afd8-d700e073bfff" (UID: "ef751334-a08e-4bd0-afd8-d700e073bfff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.917031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916580    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-home" pod "ef751334-a08e-4bd0-afd8-d700e073bfff" (UID: "ef751334-a08e-4bd0-afd8-d700e073bfff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.917031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916952    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.917031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916967    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.917031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.916977    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.918470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.918448    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-dshm" pod "ef751334-a08e-4bd0-afd8-d700e073bfff" (UID: "ef751334-a08e-4bd0-afd8-d700e073bfff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.918673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.918650    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef751334-a08e-4bd0-afd8-d700e073bfff-kube-api-access-7dcsl" pod "ef751334-a08e-4bd0-afd8-d700e073bfff" (UID: "ef751334-a08e-4bd0-afd8-d700e073bfff"). InnerVolumeSpecName "kube-api-access-7dcsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:19.918732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:19.918666    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef751334-a08e-4bd0-afd8-d700e073bfff-tls-certs" pod "ef751334-a08e-4bd0-afd8-d700e073bfff" (UID: "ef751334-a08e-4bd0-afd8-d700e073bfff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:20.018378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.018343    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dcsl\" (UniqueName: \"kubernetes.io/projected/ef751334-a08e-4bd0-afd8-d700e073bfff-kube-api-access-7dcsl\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:20.018378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.018374    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef751334-a08e-4bd0-afd8-d700e073bfff-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:20.018378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.018384    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef751334-a08e-4bd0-afd8-d700e073bfff-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:20.115915 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.115877    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dea1f43-5e0f-4c69-983a-e3c9f166d224" path="/var/lib/kubelet/pods/7dea1f43-5e0f-4c69-983a-e3c9f166d224/volumes"
Oct 30 19:44:20.786000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.785963    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" event={"ID":"441166f0-c2a7-4003-b34a-d5bc35b66b73","Type":"ContainerStarted","Data":"e277530268045f3afa17f100cd6b7298db775457d44b27a8a8eb0ca58ef89d24"}
Oct 30 19:44:20.786548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.786023    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" event={"ID":"441166f0-c2a7-4003-b34a-d5bc35b66b73","Type":"ContainerStarted","Data":"9abb66554db844b0122d91c9e67bd9a77d4ecba2255d4e6eacb3d22e1463e916"}
Oct 30 19:44:20.787904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.787873    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"
Oct 30 19:44:20.787904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.787896    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw" event={"ID":"ef751334-a08e-4bd0-afd8-d700e073bfff","Type":"ContainerDied","Data":"85f56d78e6bc1d6199711d2e4f9261c6b3f53286e5fa5359507a3b7fd27ad1e1"}
Oct 30 19:44:20.788122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.787936    6777 scope.go:122] "RemoveContainer" containerID="ca250a79893739698f92dc3331c32438020b4b16a2e546ef3bd6c054cf177d60"
Oct 30 19:44:20.809892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.809836    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" podStartSLOduration=1.809818998 podStartE2EDuration="1.809818998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:20.804880128 +0000 UTC m=+2242.801795514" watchObservedRunningTime="2026-10-30 19:44:20.809818998 +0000 UTC m=+2242.806734430"
Oct 30 19:44:20.824680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.824643    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"]
Oct 30 19:44:20.828940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:20.828905    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/rate-test-model-fd7c77-kserve-775d7d46d5-njbcw"]
Oct 30 19:44:21.901086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:21.893624    6777 generic.go:356] "Generic (PLEG): container finished" podID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerID="e277530268045f3afa17f100cd6b7298db775457d44b27a8a8eb0ca58ef89d24" exitCode=7
Oct 30 19:44:21.901086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:21.893860    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" event={"ID":"441166f0-c2a7-4003-b34a-d5bc35b66b73","Type":"ContainerDied","Data":"e277530268045f3afa17f100cd6b7298db775457d44b27a8a8eb0ca58ef89d24"}
Oct 30 19:44:21.903860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:21.901830    6777 scope.go:122] "RemoveContainer" containerID="e277530268045f3afa17f100cd6b7298db775457d44b27a8a8eb0ca58ef89d24"
Oct 30 19:44:22.284140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:22.280545    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" path="/var/lib/kubelet/pods/ef751334-a08e-4bd0-afd8-d700e073bfff/volumes"
Oct 30 19:44:22.845514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:22.845415    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:23.151038 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:23.150920    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" event={"ID":"441166f0-c2a7-4003-b34a-d5bc35b66b73","Type":"ContainerStarted","Data":"64e590bace4dde4584c02cdb1dfae840e36a80e99552e18daa04ee1f8ff76d8e"}
Oct 30 19:44:24.161400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:24.161360    6777 generic.go:356] "Generic (PLEG): container finished" podID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerID="64e590bace4dde4584c02cdb1dfae840e36a80e99552e18daa04ee1f8ff76d8e" exitCode=7
Oct 30 19:44:24.161879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:24.161442    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" event={"ID":"441166f0-c2a7-4003-b34a-d5bc35b66b73","Type":"ContainerDied","Data":"64e590bace4dde4584c02cdb1dfae840e36a80e99552e18daa04ee1f8ff76d8e"}
Oct 30 19:44:24.161879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:24.161500    6777 scope.go:122] "RemoveContainer" containerID="e277530268045f3afa17f100cd6b7298db775457d44b27a8a8eb0ca58ef89d24"
Oct 30 19:44:24.161879 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:24.161856    6777 scope.go:122] "RemoveContainer" containerID="64e590bace4dde4584c02cdb1dfae840e36a80e99552e18daa04ee1f8ff76d8e"
Oct 30 19:44:24.162334 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:24.162308    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb_odh-ai-gateway-infra(441166f0-c2a7-4003-b34a-d5bc35b66b73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73"
Oct 30 19:44:24.509423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:24.509321    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9" podUID="1299b4be-5ada-45f6-b7bb-9e0e25e6854f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:25.171883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:25.171848    6777 scope.go:122] "RemoveContainer" containerID="64e590bace4dde4584c02cdb1dfae840e36a80e99552e18daa04ee1f8ff76d8e"
Oct 30 19:44:25.172417 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:25.172390    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb_odh-ai-gateway-infra(441166f0-c2a7-4003-b34a-d5bc35b66b73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73"
Oct 30 19:44:26.631011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:26.630984    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:44:26.723430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:26.723397    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-kube-api-access-vfxrw\" (UniqueName: \"kubernetes.io/projected/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-kube-api-access-vfxrw\") pod \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\" (UID: \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\") "
Oct 30 19:44:26.723618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:26.723473    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-plugins-config-volume\") pod \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\" (UID: \"1299b4be-5ada-45f6-b7bb-9e0e25e6854f\") "
Oct 30 19:44:26.724911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:26.724872    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-plugins-config-volume" pod "1299b4be-5ada-45f6-b7bb-9e0e25e6854f" (UID: "1299b4be-5ada-45f6-b7bb-9e0e25e6854f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:26.726131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:26.726084    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-kube-api-access-vfxrw" pod "1299b4be-5ada-45f6-b7bb-9e0e25e6854f" (UID: "1299b4be-5ada-45f6-b7bb-9e0e25e6854f"). InnerVolumeSpecName "kube-api-access-vfxrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:26.826261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:26.826159    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfxrw\" (UniqueName: \"kubernetes.io/projected/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-kube-api-access-vfxrw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:26.826261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:26.826189    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1299b4be-5ada-45f6-b7bb-9e0e25e6854f-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:27.191238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:27.191147    6777 generic.go:356] "Generic (PLEG): container finished" podID="1299b4be-5ada-45f6-b7bb-9e0e25e6854f" containerID="0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81" exitCode=0
Oct 30 19:44:27.191238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:27.191198    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"
Oct 30 19:44:27.191238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:27.191227    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9" event={"ID":"1299b4be-5ada-45f6-b7bb-9e0e25e6854f","Type":"ContainerDied","Data":"0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81"}
Oct 30 19:44:27.191495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:27.191285    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9" event={"ID":"1299b4be-5ada-45f6-b7bb-9e0e25e6854f","Type":"ContainerDied","Data":"884f58e79e9fe15764d7b9a15a3b68df813aa3964368ea9d8733fcf5736f3a5e"}
Oct 30 19:44:27.191495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:27.191318    6777 scope.go:122] "RemoveContainer" containerID="0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81"
Oct 30 19:44:27.205544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:27.205518    6777 scope.go:122] "RemoveContainer" containerID="0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81"
Oct 30 19:44:27.205884 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:27.205864    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81\": container with ID starting with 0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81 not found: ID does not exist" containerID="0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81"
Oct 30 19:44:27.205934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:27.205893    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81"} err="failed to get container status \"0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81\": rpc error: code = NotFound desc = could not find container \"0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81\": container with ID starting with 0c4d8026990b6bf75078a6e931fc5caf8047cc23e05f240c0d35e51cf9b07e81 not found: ID does not exist"
Oct 30 19:44:27.225780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:27.225744    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"]
Oct 30 19:44:27.230749 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:27.230716    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bce71f-7df86d777d-qjwv9"]
Oct 30 19:44:28.115859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:28.115823    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1299b4be-5ada-45f6-b7bb-9e0e25e6854f" path="/var/lib/kubelet/pods/1299b4be-5ada-45f6-b7bb-9e0e25e6854f/volumes"
Oct 30 19:44:29.080706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:29.080670    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"]
Oct 30 19:44:34.081262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.081207    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr" podUID="d2818989-ff6b-4cba-9269-a91dc8d6a9c4" containerName="payload-pre-processing" containerID="cri-o://3aea1afc467c478b748888708dedacf472263ff7720afaa5eee7014813eef6be" gracePeriod=25
Oct 30 19:44:34.291515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.291445    6777 generic.go:356] "Generic (PLEG): container finished" podID="d2818989-ff6b-4cba-9269-a91dc8d6a9c4" containerID="3aea1afc467c478b748888708dedacf472263ff7720afaa5eee7014813eef6be" exitCode=0
Oct 30 19:44:34.292047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.292016    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr" event={"ID":"d2818989-ff6b-4cba-9269-a91dc8d6a9c4","Type":"ContainerDied","Data":"3aea1afc467c478b748888708dedacf472263ff7720afaa5eee7014813eef6be"}
Oct 30 19:44:34.457917 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.457542    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:44:34.492231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.492192    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"]
Oct 30 19:44:34.493110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.493045    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d2818989-ff6b-4cba-9269-a91dc8d6a9c4" containerName="payload-pre-processing"
Oct 30 19:44:34.493110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.493111    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2818989-ff6b-4cba-9269-a91dc8d6a9c4" containerName="payload-pre-processing"
Oct 30 19:44:34.493327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.493175    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1299b4be-5ada-45f6-b7bb-9e0e25e6854f" containerName="payload-processing"
Oct 30 19:44:34.493327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.493181    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1299b4be-5ada-45f6-b7bb-9e0e25e6854f" containerName="payload-processing"
Oct 30 19:44:34.493327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.493192    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main"
Oct 30 19:44:34.493327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.493197    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main"
Oct 30 19:44:34.493593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.493341    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2818989-ff6b-4cba-9269-a91dc8d6a9c4" containerName="payload-pre-processing"
Oct 30 19:44:34.493593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.493361    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1299b4be-5ada-45f6-b7bb-9e0e25e6854f" containerName="payload-processing"
Oct 30 19:44:34.493593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.493368    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef751334-a08e-4bd0-afd8-d700e073bfff" containerName="main"
Oct 30 19:44:34.497512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.497486    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.513571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.513544    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1a844e-openshift-default-dockercfg-465qc\""
Oct 30 19:44:34.514320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.514290    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-plugins-config-volume\") pod \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\" (UID: \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\") "
Oct 30 19:44:34.514940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.514529    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-kube-api-access-fc2s9\" (UniqueName: \"kubernetes.io/projected/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-kube-api-access-fc2s9\") pod \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\" (UID: \"d2818989-ff6b-4cba-9269-a91dc8d6a9c4\") "
Oct 30 19:44:34.516520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.516480    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"]
Oct 30 19:44:34.520293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.520267    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-plugins-config-volume" pod "d2818989-ff6b-4cba-9269-a91dc8d6a9c4" (UID: "d2818989-ff6b-4cba-9269-a91dc8d6a9c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:34.524960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.524911    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-kube-api-access-fc2s9" pod "d2818989-ff6b-4cba-9269-a91dc8d6a9c4" (UID: "d2818989-ff6b-4cba-9269-a91dc8d6a9c4"). InnerVolumeSpecName "kube-api-access-fc2s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:34.616559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.616520    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.616778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.616568    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.616778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.616589    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf87c\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-kube-api-access-zf87c\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.616909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.616810    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcfd0a26-981e-4313-b6d4-66dbf0107982-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.616980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.616918    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.617103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.617088    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.617186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.617172    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.617252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.617226    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.617341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.617325    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.617547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.617522    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc2s9\" (UniqueName: \"kubernetes.io/projected/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-kube-api-access-fc2s9\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.617547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.617548    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2818989-ff6b-4cba-9269-a91dc8d6a9c4-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.719496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719398    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.719496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719460    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.719496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719484    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf87c\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-kube-api-access-zf87c\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.719807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719555    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcfd0a26-981e-4313-b6d4-66dbf0107982-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.719807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719607    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.719807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719711    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.719807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719754    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.719807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719794    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.720741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719849    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.720741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.719848    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.720741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.720282    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.720741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.720370    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.720741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.720473    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.721146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.721033    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcfd0a26-981e-4313-b6d4-66dbf0107982-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.722461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.722438    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.722823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.722803    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.738220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.738185    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf87c\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-kube-api-access-zf87c\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.738440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.738383    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:34.847992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:34.847949    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:35.325812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:35.324790    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr" event={"ID":"d2818989-ff6b-4cba-9269-a91dc8d6a9c4","Type":"ContainerDied","Data":"40eb6c4854eecd0d77e4ee1f5459ec9b382fdaaf3108d00b00e3a36cbbd679b5"}
Oct 30 19:44:35.325812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:35.324847    6777 scope.go:122] "RemoveContainer" containerID="3aea1afc467c478b748888708dedacf472263ff7720afaa5eee7014813eef6be"
Oct 30 19:44:35.325812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:35.325005    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"
Oct 30 19:44:35.366808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:35.366736    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"]
Oct 30 19:44:35.371899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:35.371867    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-178bcc-b4dc5499-pc9sr"]
Oct 30 19:44:35.498817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:35.498785    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"]
Oct 30 19:44:35.499597 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:44:35.499571    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfd0a26_981e_4313_b6d4_66dbf0107982.slice/crio-10de4d105a405531de73c0aac966281326c262c343f4b73c195de1fe9159dbff WatchSource:0}: Error finding container 10de4d105a405531de73c0aac966281326c262c343f4b73c195de1fe9159dbff: Status 404 returned error can't find the container with id 10de4d105a405531de73c0aac966281326c262c343f4b73c195de1fe9159dbff
Oct 30 19:44:35.502860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:35.502827    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:44:35.502961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:35.502894    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:44:35.502961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:35.502922    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:44:36.105570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:36.105520    6777 scope.go:122] "RemoveContainer" containerID="64e590bace4dde4584c02cdb1dfae840e36a80e99552e18daa04ee1f8ff76d8e"
Oct 30 19:44:36.122006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:36.121967    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2818989-ff6b-4cba-9269-a91dc8d6a9c4" path="/var/lib/kubelet/pods/d2818989-ff6b-4cba-9269-a91dc8d6a9c4/volumes"
Oct 30 19:44:36.336226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:36.336182    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" event={"ID":"fcfd0a26-981e-4313-b6d4-66dbf0107982","Type":"ContainerStarted","Data":"206c54088f0a9d30ba7d4a55053462985cacc6ab41e5035ae4175334ad9b16e7"}
Oct 30 19:44:36.336682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:36.336242    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" event={"ID":"fcfd0a26-981e-4313-b6d4-66dbf0107982","Type":"ContainerStarted","Data":"10de4d105a405531de73c0aac966281326c262c343f4b73c195de1fe9159dbff"}
Oct 30 19:44:36.422002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:36.421889    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" podStartSLOduration=24.421872282 podStartE2EDuration="24.421872282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:36.418479422 +0000 UTC m=+2258.415394809" watchObservedRunningTime="2026-10-30 19:44:36.421872282 +0000 UTC m=+2258.418787666"
Oct 30 19:44:36.848611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:36.848574    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:36.855755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:36.855722    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:37.349391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:37.349354    6777 generic.go:356] "Generic (PLEG): container finished" podID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerID="5302aea5f3d17bb0c65771de0a28ca26358806b6cefd8e0d0cc6d04ee5ce74c2" exitCode=7
Oct 30 19:44:37.349920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:37.349424    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" event={"ID":"441166f0-c2a7-4003-b34a-d5bc35b66b73","Type":"ContainerDied","Data":"5302aea5f3d17bb0c65771de0a28ca26358806b6cefd8e0d0cc6d04ee5ce74c2"}
Oct 30 19:44:37.349920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:37.349511    6777 scope.go:122] "RemoveContainer" containerID="64e590bace4dde4584c02cdb1dfae840e36a80e99552e18daa04ee1f8ff76d8e"
Oct 30 19:44:37.350207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:37.350180    6777 scope.go:122] "RemoveContainer" containerID="5302aea5f3d17bb0c65771de0a28ca26358806b6cefd8e0d0cc6d04ee5ce74c2"
Oct 30 19:44:37.350685 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:37.350660    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb_odh-ai-gateway-infra(441166f0-c2a7-4003-b34a-d5bc35b66b73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73"
Oct 30 19:44:37.351014 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:37.350991    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:37.351974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:37.351957    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:44:38.385980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.385935    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"]
Oct 30 19:44:38.572794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.572766    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:38.642558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.642468    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441166f0-c2a7-4003-b34a-d5bc35b66b73-kube-api-access-nntc4\" (UniqueName: \"kubernetes.io/projected/441166f0-c2a7-4003-b34a-d5bc35b66b73-kube-api-access-nntc4\") pod \"441166f0-c2a7-4003-b34a-d5bc35b66b73\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") "
Oct 30 19:44:38.642734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.642716    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-openshift-ca\") pod \"441166f0-c2a7-4003-b34a-d5bc35b66b73\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") "
Oct 30 19:44:38.642805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.642773    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-xks-ca\") pod \"441166f0-c2a7-4003-b34a-d5bc35b66b73\" (UID: \"441166f0-c2a7-4003-b34a-d5bc35b66b73\") "
Oct 30 19:44:38.643179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.643141    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-openshift-ca" pod "441166f0-c2a7-4003-b34a-d5bc35b66b73" (UID: "441166f0-c2a7-4003-b34a-d5bc35b66b73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:38.643920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.643889    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:38.645250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.645222    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-xks-ca" pod "441166f0-c2a7-4003-b34a-d5bc35b66b73" (UID: "441166f0-c2a7-4003-b34a-d5bc35b66b73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:38.645337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.645267    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441166f0-c2a7-4003-b34a-d5bc35b66b73-kube-api-access-nntc4" pod "441166f0-c2a7-4003-b34a-d5bc35b66b73" (UID: "441166f0-c2a7-4003-b34a-d5bc35b66b73"). InnerVolumeSpecName "kube-api-access-nntc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:38.745786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.745725    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441166f0-c2a7-4003-b34a-d5bc35b66b73-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:38.745786 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:38.745758    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nntc4\" (UniqueName: \"kubernetes.io/projected/441166f0-c2a7-4003-b34a-d5bc35b66b73-kube-api-access-nntc4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:39.314283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:39.314251    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"]
Oct 30 19:44:39.322779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:39.322740    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bce71f-5ddnr"]
Oct 30 19:44:39.378274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:39.378230    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"]
Oct 30 19:44:39.378514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:39.378491    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy" containerID="cri-o://b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0" gracePeriod=30
Oct 30 19:44:39.390446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:39.390419    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"
Oct 30 19:44:39.394019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:39.393982    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb" event={"ID":"441166f0-c2a7-4003-b34a-d5bc35b66b73","Type":"ContainerDied","Data":"9abb66554db844b0122d91c9e67bd9a77d4ecba2255d4e6eacb3d22e1463e916"}
Oct 30 19:44:39.394190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:39.394072    6777 scope.go:122] "RemoveContainer" containerID="5302aea5f3d17bb0c65771de0a28ca26358806b6cefd8e0d0cc6d04ee5ce74c2"
Oct 30 19:44:39.437212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:39.437172    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"]
Oct 30 19:44:39.443180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:39.443151    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-96zhb"]
Oct 30 19:44:40.119072 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:40.119031    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" path="/var/lib/kubelet/pods/441166f0-c2a7-4003-b34a-d5bc35b66b73/volumes"
Oct 30 19:44:40.119867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:40.119845    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="905d89e6-38a1-4574-84b8-3a07922157ab" path="/var/lib/kubelet/pods/905d89e6-38a1-4574-84b8-3a07922157ab/volumes"
Oct 30 19:44:42.036902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:42.036865    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 19:44:42.037359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:42.036925    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:44:43.702650 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:43.702609    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:44:44.691679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.691646    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:44:44.707329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707293    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-socket\") pod \"3806290b-05d3-4460-8b13-eeef5668d2e4\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") "
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707404    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-data\") pod \"3806290b-05d3-4460-8b13-eeef5668d2e4\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") "
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707459    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-credential-socket\") pod \"3806290b-05d3-4460-8b13-eeef5668d2e4\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") "
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707616    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-token\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-token\") pod \"3806290b-05d3-4460-8b13-eeef5668d2e4\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") "
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707656    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-socket" pod "3806290b-05d3-4460-8b13-eeef5668d2e4" (UID: "3806290b-05d3-4460-8b13-eeef5668d2e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707709    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-envoy\") pod \"3806290b-05d3-4460-8b13-eeef5668d2e4\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") "
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707782    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-data" pod "3806290b-05d3-4460-8b13-eeef5668d2e4" (UID: "3806290b-05d3-4460-8b13-eeef5668d2e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707797    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-kube-api-access-82v7s\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-kube-api-access-82v7s\") pod \"3806290b-05d3-4460-8b13-eeef5668d2e4\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") "
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707816    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-credential-socket" pod "3806290b-05d3-4460-8b13-eeef5668d2e4" (UID: "3806290b-05d3-4460-8b13-eeef5668d2e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707866    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-certs\") pod \"3806290b-05d3-4460-8b13-eeef5668d2e4\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") "
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707961    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-podinfo\") pod \"3806290b-05d3-4460-8b13-eeef5668d2e4\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") "
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.707988    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3806290b-05d3-4460-8b13-eeef5668d2e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3806290b-05d3-4460-8b13-eeef5668d2e4-istiod-ca-cert\") pod \"3806290b-05d3-4460-8b13-eeef5668d2e4\" (UID: \"3806290b-05d3-4460-8b13-eeef5668d2e4\") "
Oct 30 19:44:44.708743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.708338    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-certs" pod "3806290b-05d3-4460-8b13-eeef5668d2e4" (UID: "3806290b-05d3-4460-8b13-eeef5668d2e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:44.710425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.708954    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3806290b-05d3-4460-8b13-eeef5668d2e4-istiod-ca-cert" pod "3806290b-05d3-4460-8b13-eeef5668d2e4" (UID: "3806290b-05d3-4460-8b13-eeef5668d2e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:44.710425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.709665    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:44.710425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.709691    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:44.710425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.709702    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:44.710425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.709711    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3806290b-05d3-4460-8b13-eeef5668d2e4-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:44.710425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.709721    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:44.710783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.710761    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-kube-api-access-82v7s" pod "3806290b-05d3-4460-8b13-eeef5668d2e4" (UID: "3806290b-05d3-4460-8b13-eeef5668d2e4"). InnerVolumeSpecName "kube-api-access-82v7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:44.711115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.711053    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-podinfo" pod "3806290b-05d3-4460-8b13-eeef5668d2e4" (UID: "3806290b-05d3-4460-8b13-eeef5668d2e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:44.711619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.711597    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-token" pod "3806290b-05d3-4460-8b13-eeef5668d2e4" (UID: "3806290b-05d3-4460-8b13-eeef5668d2e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:44.712344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.712319    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-envoy" pod "3806290b-05d3-4460-8b13-eeef5668d2e4" (UID: "3806290b-05d3-4460-8b13-eeef5668d2e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:44.742266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.742224    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"]
Oct 30 19:44:44.743560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743529    6777 cpu_manager.go:414] "containerMap: removing container" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerName="revoke-keys"
Oct 30 19:44:44.743560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743559    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerName="revoke-keys"
Oct 30 19:44:44.743783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743579    6777 cpu_manager.go:414] "containerMap: removing container" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy"
Oct 30 19:44:44.743783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743589    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy"
Oct 30 19:44:44.743783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743628    6777 cpu_manager.go:414] "containerMap: removing container" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerName="revoke-keys"
Oct 30 19:44:44.743783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743638    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerName="revoke-keys"
Oct 30 19:44:44.743783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743700    6777 cpu_manager.go:414] "containerMap: removing container" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy"
Oct 30 19:44:44.743783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743708    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy"
Oct 30 19:44:44.744131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743897    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerName="revoke-keys"
Oct 30 19:44:44.744131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743915    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy"
Oct 30 19:44:44.744131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743928    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerName="revoke-keys"
Oct 30 19:44:44.744131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743940    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerName="revoke-keys"
Oct 30 19:44:44.744131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.743962    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerName="istio-proxy"
Oct 30 19:44:44.750329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.750305    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.756286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.755972    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c2b092-openshift-default-dockercfg-8hfcl\""
Oct 30 19:44:44.759533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.759491    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"]
Oct 30 19:44:44.811191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.811152    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.811384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.811207    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.811431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.811407    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85923a0f-3daa-4295-9a1d-3ca15689f158-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.811469 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.811451    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.811605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.811592    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.811647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.811630    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grbw\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-kube-api-access-8grbw\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.811801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.811778    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.811848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.811834    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.812368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.811879    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.812368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.812025    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:44.812368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.812037    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:44.812368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.812049    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82v7s\" (UniqueName: \"kubernetes.io/projected/3806290b-05d3-4460-8b13-eeef5668d2e4-kube-api-access-82v7s\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:44.812368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.812073    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3806290b-05d3-4460-8b13-eeef5668d2e4-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:44.842467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.842429    6777 generic.go:356] "Generic (PLEG): container finished" podID="3806290b-05d3-4460-8b13-eeef5668d2e4" containerID="b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0" exitCode=0
Oct 30 19:44:44.842660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.842506    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"
Oct 30 19:44:44.842660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.842512    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" event={"ID":"3806290b-05d3-4460-8b13-eeef5668d2e4","Type":"ContainerDied","Data":"b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0"}
Oct 30 19:44:44.842660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.842580    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx" event={"ID":"3806290b-05d3-4460-8b13-eeef5668d2e4","Type":"ContainerDied","Data":"0bc0d4f048c803cfbeec5ec3747c97e1e20e52d45f0041ee5293c2864ff4ac66"}
Oct 30 19:44:44.842660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.842609    6777 scope.go:122] "RemoveContainer" containerID="b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0"
Oct 30 19:44:44.855540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.855517    6777 scope.go:122] "RemoveContainer" containerID="d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce"
Oct 30 19:44:44.867932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.867902    6777 scope.go:122] "RemoveContainer" containerID="b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0"
Oct 30 19:44:44.868310 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:44.868286    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0\": container with ID starting with b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0 not found: ID does not exist" containerID="b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0"
Oct 30 19:44:44.868382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.868321    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0"} err="failed to get container status \"b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0\": rpc error: code = NotFound desc = could not find container \"b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0\": container with ID starting with b94e2d01be9e8ea347d0cdd973e3374580bf8613b5102f87c05aeec43f1900e0 not found: ID does not exist"
Oct 30 19:44:44.868382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.868343    6777 scope.go:122] "RemoveContainer" containerID="d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce"
Oct 30 19:44:44.868634 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:44.868615    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce\": container with ID starting with d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce not found: ID does not exist" containerID="d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce"
Oct 30 19:44:44.868693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.868644    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce"} err="failed to get container status \"d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce\": rpc error: code = NotFound desc = could not find container \"d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce\": container with ID starting with d8fa8003fe83e7369838a37549af57466cbfa63e85127bfcd879d18681882fce not found: ID does not exist"
Oct 30 19:44:44.872612 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.872584    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"]
Oct 30 19:44:44.876855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.876827    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-178bcc-openshift-default-86ddcdb5b9-h7swx"]
Oct 30 19:44:44.924148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.915107    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.924148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.915189    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.924148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.915226    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.924148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.915290    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.924148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.915327    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.924148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.915481    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85923a0f-3daa-4295-9a1d-3ca15689f158-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.924148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.915521    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.924148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.915696    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.924148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.915750    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grbw\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-kube-api-access-8grbw\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.936083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.929233    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.936083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.929603    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.936083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.929626    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.936083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.930096    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.936083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.931515    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85923a0f-3daa-4295-9a1d-3ca15689f158-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.936083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.936083    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.937910 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.937883    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.948353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.945319    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:44.948353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:44.945536    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grbw\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-kube-api-access-8grbw\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:45.080192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:45.080158    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:45.373386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:45.373352    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"]
Oct 30 19:44:45.373812 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:44:45.373781    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85923a0f_3daa_4295_9a1d_3ca15689f158.slice/crio-48133218ca05a5b211555ebfef19bfa8f9a13c5fe7fb5ec4bae69e8130d2cfa1 WatchSource:0}: Error finding container 48133218ca05a5b211555ebfef19bfa8f9a13c5fe7fb5ec4bae69e8130d2cfa1: Status 404 returned error can't find the container with id 48133218ca05a5b211555ebfef19bfa8f9a13c5fe7fb5ec4bae69e8130d2cfa1
Oct 30 19:44:45.377310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:45.377245    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:44:45.377453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:45.377359    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:44:45.377453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:45.377439    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:44:45.858875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:45.858837    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" event={"ID":"85923a0f-3daa-4295-9a1d-3ca15689f158","Type":"ContainerStarted","Data":"94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549"}
Oct 30 19:44:45.859785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:45.858909    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" event={"ID":"85923a0f-3daa-4295-9a1d-3ca15689f158","Type":"ContainerStarted","Data":"48133218ca05a5b211555ebfef19bfa8f9a13c5fe7fb5ec4bae69e8130d2cfa1"}
Oct 30 19:44:45.887452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:45.887399    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podStartSLOduration=3.887382641 podStartE2EDuration="3.887382641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:45.884782232 +0000 UTC m=+2267.881697617" watchObservedRunningTime="2026-10-30 19:44:45.887382641 +0000 UTC m=+2267.884298025"
Oct 30 19:44:46.081713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:46.081656    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:46.088127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:46.088048    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:46.117358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:46.117278    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3806290b-05d3-4460-8b13-eeef5668d2e4" path="/var/lib/kubelet/pods/3806290b-05d3-4460-8b13-eeef5668d2e4/volumes"
Oct 30 19:44:46.871315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:46.871283    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:46.872578 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:46.872555    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:44:48.009809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:48.009770    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"]
Oct 30 19:44:48.010367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:48.010003    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy" containerID="cri-o://cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713" gracePeriod=30
Oct 30 19:44:53.381102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.381027    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:44:53.419586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.419547    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"]
Oct 30 19:44:53.420872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.420837    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy"
Oct 30 19:44:53.420872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.420867    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy"
Oct 30 19:44:53.421136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.420888    6777 cpu_manager.go:414] "containerMap: removing container" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerName="revoke-keys"
Oct 30 19:44:53.421136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.420896    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441166f0-c2a7-4003-b34a-d5bc35b66b73" containerName="revoke-keys"
Oct 30 19:44:53.421136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.421123    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerName="istio-proxy"
Oct 30 19:44:53.463629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.463236    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"]
Oct 30 19:44:53.465291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.465223    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:44:53.470434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.470405    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c2b092-dockercfg-6qp4c\""
Oct 30 19:44:53.470758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.470716    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c2b092\""
Oct 30 19:44:53.477270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.477241    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"]
Oct 30 19:44:53.477409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.477285    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"]
Oct 30 19:44:53.477409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.477358    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:44:53.543307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.543266    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-token\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-token\") pod \"b7a0e32e-00a4-4186-b20b-da060900cc17\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") "
Oct 30 19:44:53.543736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.543710    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-data\") pod \"b7a0e32e-00a4-4186-b20b-da060900cc17\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") "
Oct 30 19:44:53.543940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.543913    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-podinfo\") pod \"b7a0e32e-00a4-4186-b20b-da060900cc17\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") "
Oct 30 19:44:53.544137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.544033    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-socket\") pod \"b7a0e32e-00a4-4186-b20b-da060900cc17\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") "
Oct 30 19:44:53.544137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.544126    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-envoy\") pod \"b7a0e32e-00a4-4186-b20b-da060900cc17\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") "
Oct 30 19:44:53.544314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.544173    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-certs\") pod \"b7a0e32e-00a4-4186-b20b-da060900cc17\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") "
Oct 30 19:44:53.544314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.544245    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-kube-api-access-p66tt\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-kube-api-access-p66tt\") pod \"b7a0e32e-00a4-4186-b20b-da060900cc17\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") "
Oct 30 19:44:53.544478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.544434    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a0e32e-00a4-4186-b20b-da060900cc17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a0e32e-00a4-4186-b20b-da060900cc17-istiod-ca-cert\") pod \"b7a0e32e-00a4-4186-b20b-da060900cc17\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") "
Oct 30 19:44:53.544552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.544492    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-credential-socket\") pod \"b7a0e32e-00a4-4186-b20b-da060900cc17\" (UID: \"b7a0e32e-00a4-4186-b20b-da060900cc17\") "
Oct 30 19:44:53.545656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.545624    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-credential-socket" pod "b7a0e32e-00a4-4186-b20b-da060900cc17" (UID: "b7a0e32e-00a4-4186-b20b-da060900cc17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.546614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.546561    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plscx\" (UniqueName: \"kubernetes.io/projected/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-kube-api-access-plscx\") pod \"payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2\" (UID: \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:44:53.546958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.546936    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2\" (UID: \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:44:53.547163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.547137    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.547262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.547219    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-certs" pod "b7a0e32e-00a4-4186-b20b-da060900cc17" (UID: "b7a0e32e-00a4-4186-b20b-da060900cc17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.547351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.547287    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-socket" pod "b7a0e32e-00a4-4186-b20b-da060900cc17" (UID: "b7a0e32e-00a4-4186-b20b-da060900cc17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.547351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.547341    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-data" pod "b7a0e32e-00a4-4186-b20b-da060900cc17" (UID: "b7a0e32e-00a4-4186-b20b-da060900cc17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.548320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.548265    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a0e32e-00a4-4186-b20b-da060900cc17-istiod-ca-cert" pod "b7a0e32e-00a4-4186-b20b-da060900cc17" (UID: "b7a0e32e-00a4-4186-b20b-da060900cc17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:53.554505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.554476    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-podinfo" pod "b7a0e32e-00a4-4186-b20b-da060900cc17" (UID: "b7a0e32e-00a4-4186-b20b-da060900cc17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:53.554657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.554517    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-token" pod "b7a0e32e-00a4-4186-b20b-da060900cc17" (UID: "b7a0e32e-00a4-4186-b20b-da060900cc17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:53.558689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.558651    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-envoy" pod "b7a0e32e-00a4-4186-b20b-da060900cc17" (UID: "b7a0e32e-00a4-4186-b20b-da060900cc17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.559651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.559618    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-kube-api-access-p66tt" pod "b7a0e32e-00a4-4186-b20b-da060900cc17" (UID: "b7a0e32e-00a4-4186-b20b-da060900cc17"). InnerVolumeSpecName "kube-api-access-p66tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:53.648877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.648833    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8rk\" (UniqueName: \"kubernetes.io/projected/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-kube-api-access-tt8rk\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2\" (UID: \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:44:53.649096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.648907    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2\" (UID: \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:44:53.649096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649048    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plscx\" (UniqueName: \"kubernetes.io/projected/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-kube-api-access-plscx\") pod \"payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2\" (UID: \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:44:53.649413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649389    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2\" (UID: \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:44:53.649634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649616    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.649680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649642    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.649680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649658    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.649680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649672    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.649680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649687    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a0e32e-00a4-4186-b20b-da060900cc17-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.649865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649701    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66tt\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-kube-api-access-p66tt\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.649865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649716    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a0e32e-00a4-4186-b20b-da060900cc17-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.649865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.649730    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a0e32e-00a4-4186-b20b-da060900cc17-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.650142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.650123    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2\" (UID: \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:44:53.662541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.662505    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plscx\" (UniqueName: \"kubernetes.io/projected/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-kube-api-access-plscx\") pod \"payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2\" (UID: \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:44:53.751460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.751418    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8rk\" (UniqueName: \"kubernetes.io/projected/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-kube-api-access-tt8rk\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2\" (UID: \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:44:53.751652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.751494    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2\" (UID: \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:44:53.752327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.752305    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2\" (UID: \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:44:53.764865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.764832    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8rk\" (UniqueName: \"kubernetes.io/projected/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-kube-api-access-tt8rk\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2\" (UID: \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:44:53.796422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.796387    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:44:53.812428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.812399    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:44:53.945990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.945951    6777 generic.go:356] "Generic (PLEG): container finished" podID="b7a0e32e-00a4-4186-b20b-da060900cc17" containerID="cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713" exitCode=0
Oct 30 19:44:53.946794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.946222    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" event={"ID":"b7a0e32e-00a4-4186-b20b-da060900cc17","Type":"ContainerDied","Data":"cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713"}
Oct 30 19:44:53.946794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.946293    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t" event={"ID":"b7a0e32e-00a4-4186-b20b-da060900cc17","Type":"ContainerDied","Data":"5b634be6685b677780d05eaeea0319b873c14d40341f966134e2c9c8c2ce0b17"}
Oct 30 19:44:53.946794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.946330    6777 scope.go:122] "RemoveContainer" containerID="cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713"
Oct 30 19:44:53.946794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.946496    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"
Oct 30 19:44:53.978865 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.978824    6777 scope.go:122] "RemoveContainer" containerID="cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713"
Oct 30 19:44:53.979473 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:44:53.979441    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713\": container with ID starting with cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713 not found: ID does not exist" containerID="cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713"
Oct 30 19:44:53.979666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.979641    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713"} err="failed to get container status \"cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713\": rpc error: code = NotFound desc = could not find container \"cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713\": container with ID starting with cefaae5e3fb23280acfd9d161a9a20a544fef61514fd826de4658273a0522713 not found: ID does not exist"
Oct 30 19:44:53.988822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.988792    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"]
Oct 30 19:44:53.995892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:53.995855    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-txb2t"]
Oct 30 19:44:54.117261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.117226    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a0e32e-00a4-4186-b20b-da060900cc17" path="/var/lib/kubelet/pods/b7a0e32e-00a4-4186-b20b-da060900cc17/volumes"
Oct 30 19:44:54.152308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.152274    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"]
Oct 30 19:44:54.152593 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:44:54.152562    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13176ae0_fa1c_4020_b9aa_cbdbc28ac07f.slice/crio-a863869c49777dc65e517b0271a3a8f08a9e24d6d8e31f715da01f2183cd4ad1 WatchSource:0}: Error finding container a863869c49777dc65e517b0271a3a8f08a9e24d6d8e31f715da01f2183cd4ad1: Status 404 returned error can't find the container with id a863869c49777dc65e517b0271a3a8f08a9e24d6d8e31f715da01f2183cd4ad1
Oct 30 19:44:54.168672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.168627    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"]
Oct 30 19:44:54.177781 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:44:54.177749    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98bf8ff_e8c2_4600_ad27_06fd8e3d0bcd.slice/crio-fe44134c149d1773a91a9fa646220cbd0a25543102050ee129112ec0c9aa1e21 WatchSource:0}: Error finding container fe44134c149d1773a91a9fa646220cbd0a25543102050ee129112ec0c9aa1e21: Status 404 returned error can't find the container with id fe44134c149d1773a91a9fa646220cbd0a25543102050ee129112ec0c9aa1e21
Oct 30 19:44:54.959112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.959071    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2" event={"ID":"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd","Type":"ContainerStarted","Data":"7483689ed74efc540190fa26fb34964892eda57755866c5a1ede53d4ae3b9059"}
Oct 30 19:44:54.959637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.959144    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:44:54.959637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.959167    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2" event={"ID":"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd","Type":"ContainerStarted","Data":"fe44134c149d1773a91a9fa646220cbd0a25543102050ee129112ec0c9aa1e21"}
Oct 30 19:44:54.961226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.961196    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2" event={"ID":"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f","Type":"ContainerStarted","Data":"8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e"}
Oct 30 19:44:54.961343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.961243    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2" event={"ID":"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f","Type":"ContainerStarted","Data":"a863869c49777dc65e517b0271a3a8f08a9e24d6d8e31f715da01f2183cd4ad1"}
Oct 30 19:44:54.961343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.961269    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:44:54.980698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:54.980641    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2" podStartSLOduration=5.980625817 podStartE2EDuration="5.980625817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:54.979247683 +0000 UTC m=+2276.976163069" watchObservedRunningTime="2026-10-30 19:44:54.980625817 +0000 UTC m=+2276.977541201"
Oct 30 19:44:55.003509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:55.003452    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2" podStartSLOduration=6.00343723 podStartE2EDuration="6.00343723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:55.00111609 +0000 UTC m=+2276.998031475" watchObservedRunningTime="2026-10-30 19:44:55.00343723 +0000 UTC m=+2277.000352614"
Oct 30 19:44:55.079297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:55.079258    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"]
Oct 30 19:44:55.086313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:55.086272    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e273b0d-lh4hc"]
Oct 30 19:44:56.117462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:44:56.117427    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ffca28b-019b-454e-9681-81fc0de7c912" path="/var/lib/kubelet/pods/7ffca28b-019b-454e-9681-81fc0de7c912/volumes"
Oct 30 19:45:00.149905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.149865    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"]
Oct 30 19:45:00.157146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.157109    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:00.158673 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.158435    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"]
Oct 30 19:45:00.187509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.187477    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"]
Oct 30 19:45:00.188632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.188602    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:00.196473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.196415    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"]
Oct 30 19:45:00.196840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.196817    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:00.207440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.207407    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"]
Oct 30 19:45:00.207902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.207563    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:00.229705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.228870    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"]
Oct 30 19:45:00.240708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.240680    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"]
Oct 30 19:45:00.246747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.246720    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"]
Oct 30 19:45:00.256238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.256211    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:45:00.284993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.284947    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe76d57-03af-4460-b0ca-84ab80ef060b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc\" (UID: \"6fe76d57-03af-4460-b0ca-84ab80ef060b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:00.285503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.285479    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck74g\" (UniqueName: \"kubernetes.io/projected/6fe76d57-03af-4460-b0ca-84ab80ef060b-kube-api-access-ck74g\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc\" (UID: \"6fe76d57-03af-4460-b0ca-84ab80ef060b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:00.387574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.387529    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debc150f-bdfc-401e-876f-f110044f94b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z\" (UID: \"debc150f-bdfc-401e-876f-f110044f94b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:00.387812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.387578    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l58k\" (UniqueName: \"kubernetes.io/projected/debc150f-bdfc-401e-876f-f110044f94b4-kube-api-access-7l58k\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z\" (UID: \"debc150f-bdfc-401e-876f-f110044f94b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:00.387812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.387745    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab87e647-dc82-4dbf-b754-45e414878aeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2\" (UID: \"ab87e647-dc82-4dbf-b754-45e414878aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:00.387812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.387796    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck74g\" (UniqueName: \"kubernetes.io/projected/6fe76d57-03af-4460-b0ca-84ab80ef060b-kube-api-access-ck74g\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc\" (UID: \"6fe76d57-03af-4460-b0ca-84ab80ef060b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:00.387998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.387901    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5k6\" (UniqueName: \"kubernetes.io/projected/694b1588-e272-4115-ae41-908654b72649-kube-api-access-gg5k6\") pod \"maas-api-key-cleanup-29889825-8t28c\" (UID: \"694b1588-e272-4115-ae41-908654b72649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:00.388077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.388007    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe76d57-03af-4460-b0ca-84ab80ef060b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc\" (UID: \"6fe76d57-03af-4460-b0ca-84ab80ef060b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:00.388173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.388150    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/694b1588-e272-4115-ae41-908654b72649-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-8t28c\" (UID: \"694b1588-e272-4115-ae41-908654b72649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:00.388313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.388295    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvj55\" (UniqueName: \"kubernetes.io/projected/ab87e647-dc82-4dbf-b754-45e414878aeb-kube-api-access-rvj55\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2\" (UID: \"ab87e647-dc82-4dbf-b754-45e414878aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:00.388694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.388674    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe76d57-03af-4460-b0ca-84ab80ef060b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc\" (UID: \"6fe76d57-03af-4460-b0ca-84ab80ef060b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:00.400471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.400390    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck74g\" (UniqueName: \"kubernetes.io/projected/6fe76d57-03af-4460-b0ca-84ab80ef060b-kube-api-access-ck74g\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc\" (UID: \"6fe76d57-03af-4460-b0ca-84ab80ef060b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:00.489908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.489858    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/694b1588-e272-4115-ae41-908654b72649-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-8t28c\" (UID: \"694b1588-e272-4115-ae41-908654b72649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:00.490187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.490002    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvj55\" (UniqueName: \"kubernetes.io/projected/ab87e647-dc82-4dbf-b754-45e414878aeb-kube-api-access-rvj55\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2\" (UID: \"ab87e647-dc82-4dbf-b754-45e414878aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:00.490273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.490235    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debc150f-bdfc-401e-876f-f110044f94b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z\" (UID: \"debc150f-bdfc-401e-876f-f110044f94b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:00.490340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.490266    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l58k\" (UniqueName: \"kubernetes.io/projected/debc150f-bdfc-401e-876f-f110044f94b4-kube-api-access-7l58k\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z\" (UID: \"debc150f-bdfc-401e-876f-f110044f94b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:00.490396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.490353    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab87e647-dc82-4dbf-b754-45e414878aeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2\" (UID: \"ab87e647-dc82-4dbf-b754-45e414878aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:00.490451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.490417    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5k6\" (UniqueName: \"kubernetes.io/projected/694b1588-e272-4115-ae41-908654b72649-kube-api-access-gg5k6\") pod \"maas-api-key-cleanup-29889825-8t28c\" (UID: \"694b1588-e272-4115-ae41-908654b72649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:00.490766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.490742    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/694b1588-e272-4115-ae41-908654b72649-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-8t28c\" (UID: \"694b1588-e272-4115-ae41-908654b72649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:00.490906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.490887    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debc150f-bdfc-401e-876f-f110044f94b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z\" (UID: \"debc150f-bdfc-401e-876f-f110044f94b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:00.491016 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.490991    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab87e647-dc82-4dbf-b754-45e414878aeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2\" (UID: \"ab87e647-dc82-4dbf-b754-45e414878aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:00.503217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.503186    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l58k\" (UniqueName: \"kubernetes.io/projected/debc150f-bdfc-401e-876f-f110044f94b4-kube-api-access-7l58k\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z\" (UID: \"debc150f-bdfc-401e-876f-f110044f94b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:00.503372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.503276    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5k6\" (UniqueName: \"kubernetes.io/projected/694b1588-e272-4115-ae41-908654b72649-kube-api-access-gg5k6\") pod \"maas-api-key-cleanup-29889825-8t28c\" (UID: \"694b1588-e272-4115-ae41-908654b72649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:00.503372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.503284    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvj55\" (UniqueName: \"kubernetes.io/projected/ab87e647-dc82-4dbf-b754-45e414878aeb-kube-api-access-rvj55\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2\" (UID: \"ab87e647-dc82-4dbf-b754-45e414878aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:00.514069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.514029    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:00.515880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.515857    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:00.532955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.532925    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:00.540550 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.540518    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:00.949402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:00.949369    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"]
Oct 30 19:45:00.949935 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:00.949905    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694b1588_e272_4115_ae41_908654b72649.slice/crio-2a1d7e7dbab58f3154da0ae24f38324e715dacddf4145a9ed9ea9dd51b8355cd WatchSource:0}: Error finding container 2a1d7e7dbab58f3154da0ae24f38324e715dacddf4145a9ed9ea9dd51b8355cd: Status 404 returned error can't find the container with id 2a1d7e7dbab58f3154da0ae24f38324e715dacddf4145a9ed9ea9dd51b8355cd
Oct 30 19:45:01.022535 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:01.022504    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe76d57_03af_4460_b0ca_84ab80ef060b.slice/crio-84a492c5c5d214cfab90dd2ba6d5c1130fb07cdc50e524df8648c189663f306d WatchSource:0}: Error finding container 84a492c5c5d214cfab90dd2ba6d5c1130fb07cdc50e524df8648c189663f306d: Status 404 returned error can't find the container with id 84a492c5c5d214cfab90dd2ba6d5c1130fb07cdc50e524df8648c189663f306d
Oct 30 19:45:01.023758 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:01.023733    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebc150f_bdfc_401e_876f_f110044f94b4.slice/crio-ad7b68f6d3987b6f38a6c18c5fe766686a08c608e1350e57632bcd79b2e284ea WatchSource:0}: Error finding container ad7b68f6d3987b6f38a6c18c5fe766686a08c608e1350e57632bcd79b2e284ea: Status 404 returned error can't find the container with id ad7b68f6d3987b6f38a6c18c5fe766686a08c608e1350e57632bcd79b2e284ea
Oct 30 19:45:01.024488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:01.024469    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"]
Oct 30 19:45:01.029509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:01.029483    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"]
Oct 30 19:45:01.068660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:01.068626    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"]
Oct 30 19:45:01.074909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:01.074881    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" event={"ID":"694b1588-e272-4115-ae41-908654b72649","Type":"ContainerStarted","Data":"2a1d7e7dbab58f3154da0ae24f38324e715dacddf4145a9ed9ea9dd51b8355cd"}
Oct 30 19:45:01.076444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:01.076421    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" event={"ID":"6fe76d57-03af-4460-b0ca-84ab80ef060b","Type":"ContainerStarted","Data":"84a492c5c5d214cfab90dd2ba6d5c1130fb07cdc50e524df8648c189663f306d"}
Oct 30 19:45:01.077846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:01.077824    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" event={"ID":"debc150f-bdfc-401e-876f-f110044f94b4","Type":"ContainerStarted","Data":"ad7b68f6d3987b6f38a6c18c5fe766686a08c608e1350e57632bcd79b2e284ea"}
Oct 30 19:45:01.995658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:01.995558    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:45:02.093807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:02.093764    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" event={"ID":"694b1588-e272-4115-ae41-908654b72649","Type":"ContainerStarted","Data":"71028a4a98074b39217c7012f2f8665861594c48c8b737a2281147fcfa5e7111"}
Oct 30 19:45:02.096624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:02.096588    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" event={"ID":"ab87e647-dc82-4dbf-b754-45e414878aeb","Type":"ContainerStarted","Data":"f8d987a94a6ea9e5b067fecc3b5d6305615495b58eeac850eca520a61c6c12d8"}
Oct 30 19:45:02.096806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:02.096649    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" event={"ID":"ab87e647-dc82-4dbf-b754-45e414878aeb","Type":"ContainerStarted","Data":"a3c45278de65eba53c7b82be734c7eb9d609cfc8586d1f909995bd429f8e771d"}
Oct 30 19:45:02.098602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:02.098573    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" event={"ID":"6fe76d57-03af-4460-b0ca-84ab80ef060b","Type":"ContainerStarted","Data":"0aeb966ffc886bdf9533e6038b1faee8b5c30c837aae7bf2b8ac30b9021fd181"}
Oct 30 19:45:02.100472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:02.100445    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" event={"ID":"debc150f-bdfc-401e-876f-f110044f94b4","Type":"ContainerStarted","Data":"51951878c936cf751a004fc4edd5abc767dfc33d21ad1a57c2fd096e044023d3"}
Oct 30 19:45:02.117714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:02.117660    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" podStartSLOduration=2.117645311 podStartE2EDuration="2.117645311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.116350645 +0000 UTC m=+2284.113266029" watchObservedRunningTime="2026-10-30 19:45:02.117645311 +0000 UTC m=+2284.114560692"
Oct 30 19:45:02.137683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:02.137620    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" podStartSLOduration=2.137607488 podStartE2EDuration="2.137607488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.136723297 +0000 UTC m=+2284.133638684" watchObservedRunningTime="2026-10-30 19:45:02.137607488 +0000 UTC m=+2284.134522871"
Oct 30 19:45:02.171163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:02.166171    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" podStartSLOduration=2.166129717 podStartE2EDuration="2.166129717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.155850173 +0000 UTC m=+2284.152765558" watchObservedRunningTime="2026-10-30 19:45:02.166129717 +0000 UTC m=+2284.163045102"
Oct 30 19:45:02.197431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:02.197376    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" podStartSLOduration=2.197357717 podStartE2EDuration="2.197357717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.194444862 +0000 UTC m=+2284.191360246" watchObservedRunningTime="2026-10-30 19:45:02.197357717 +0000 UTC m=+2284.194273160"
Oct 30 19:45:03.029854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.029818    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"]
Oct 30 19:45:03.030397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.030099    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx" podUID="a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" containerName="maas-api" containerID="cri-o://e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e" gracePeriod=30
Oct 30 19:45:03.115410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.112228    6777 generic.go:356] "Generic (PLEG): container finished" podID="694b1588-e272-4115-ae41-908654b72649" containerID="71028a4a98074b39217c7012f2f8665861594c48c8b737a2281147fcfa5e7111" exitCode=60
Oct 30 19:45:03.115410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.112390    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" event={"ID":"694b1588-e272-4115-ae41-908654b72649","Type":"ContainerDied","Data":"71028a4a98074b39217c7012f2f8665861594c48c8b737a2281147fcfa5e7111"}
Oct 30 19:45:03.115410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.112915    6777 scope.go:122] "RemoveContainer" containerID="71028a4a98074b39217c7012f2f8665861594c48c8b737a2281147fcfa5e7111"
Oct 30 19:45:03.116450 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.116421    6777 generic.go:356] "Generic (PLEG): container finished" podID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerID="f8d987a94a6ea9e5b067fecc3b5d6305615495b58eeac850eca520a61c6c12d8" exitCode=60
Oct 30 19:45:03.116588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.116548    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" event={"ID":"ab87e647-dc82-4dbf-b754-45e414878aeb","Type":"ContainerDied","Data":"f8d987a94a6ea9e5b067fecc3b5d6305615495b58eeac850eca520a61c6c12d8"}
Oct 30 19:45:03.116917 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.116896    6777 scope.go:122] "RemoveContainer" containerID="f8d987a94a6ea9e5b067fecc3b5d6305615495b58eeac850eca520a61c6c12d8"
Oct 30 19:45:03.131895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.131695    6777 generic.go:356] "Generic (PLEG): container finished" podID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerID="0aeb966ffc886bdf9533e6038b1faee8b5c30c837aae7bf2b8ac30b9021fd181" exitCode=60
Oct 30 19:45:03.132445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.132394    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" event={"ID":"6fe76d57-03af-4460-b0ca-84ab80ef060b","Type":"ContainerDied","Data":"0aeb966ffc886bdf9533e6038b1faee8b5c30c837aae7bf2b8ac30b9021fd181"}
Oct 30 19:45:03.134586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.133625    6777 scope.go:122] "RemoveContainer" containerID="0aeb966ffc886bdf9533e6038b1faee8b5c30c837aae7bf2b8ac30b9021fd181"
Oct 30 19:45:03.161238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.161198    6777 generic.go:356] "Generic (PLEG): container finished" podID="debc150f-bdfc-401e-876f-f110044f94b4" containerID="51951878c936cf751a004fc4edd5abc767dfc33d21ad1a57c2fd096e044023d3" exitCode=60
Oct 30 19:45:03.161519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.161500    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" event={"ID":"debc150f-bdfc-401e-876f-f110044f94b4","Type":"ContainerDied","Data":"51951878c936cf751a004fc4edd5abc767dfc33d21ad1a57c2fd096e044023d3"}
Oct 30 19:45:03.162341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.162314    6777 scope.go:122] "RemoveContainer" containerID="51951878c936cf751a004fc4edd5abc767dfc33d21ad1a57c2fd096e044023d3"
Oct 30 19:45:03.183193 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.183152    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"]
Oct 30 19:45:03.183552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.183513    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss" podUID="fc642a52-d707-409a-9529-a0e843a06c66" containerName="payload-pre-processing" containerID="cri-o://d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5" gracePeriod=30
Oct 30 19:45:03.252139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.252044    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"]
Oct 30 19:45:03.253645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.253611    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy" containerID="cri-o://e6958c58414ce6b7c6cb5a040b603f4ca1460a747e28e8e8d72b821afb6dad45" gracePeriod=30
Oct 30 19:45:03.882764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.882734    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:45:03.981051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.981016    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"]
Oct 30 19:45:03.988145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.982941    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" containerName="maas-api"
Oct 30 19:45:03.988145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.982967    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" containerName="maas-api"
Oct 30 19:45:03.988145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:03.983252    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" containerName="maas-api"
Oct 30 19:45:04.011243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.011213    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"]
Oct 30 19:45:04.014680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.014217    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:04.019530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.019504    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"]
Oct 30 19:45:04.019777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.019614    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:04.026163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.026079    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:45:04.027540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.027444    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"]
Oct 30 19:45:04.027932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.027912    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:04.028599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.028572    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fc642a52-d707-409a-9529-a0e843a06c66" containerName="payload-pre-processing"
Oct 30 19:45:04.028599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.028598    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc642a52-d707-409a-9529-a0e843a06c66" containerName="payload-pre-processing"
Oct 30 19:45:04.028822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.028804    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc642a52-d707-409a-9529-a0e843a06c66" containerName="payload-pre-processing"
Oct 30 19:45:04.033175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.033130    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"]
Oct 30 19:45:04.034274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.034253    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:04.038670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.038599    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"]
Oct 30 19:45:04.038803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.038766    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"]
Oct 30 19:45:04.038803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.038706    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:04.073213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.073182    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"]
Oct 30 19:45:04.073748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.073722    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-metrics-certs\") pod \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") "
Oct 30 19:45:04.073850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.073822    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc642a52-d707-409a-9529-a0e843a06c66-kube-api-access-sdjn2\" (UniqueName: \"kubernetes.io/projected/fc642a52-d707-409a-9529-a0e843a06c66-kube-api-access-sdjn2\") pod \"fc642a52-d707-409a-9529-a0e843a06c66\" (UID: \"fc642a52-d707-409a-9529-a0e843a06c66\") "
Oct 30 19:45:04.074631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.073868    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc642a52-d707-409a-9529-a0e843a06c66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc642a52-d707-409a-9529-a0e843a06c66-plugins-config-volume\") pod \"fc642a52-d707-409a-9529-a0e843a06c66\" (UID: \"fc642a52-d707-409a-9529-a0e843a06c66\") "
Oct 30 19:45:04.074631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.074029    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-maas-api-tls\") pod \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") "
Oct 30 19:45:04.074631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.074277    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-kube-api-access-66rsr\" (UniqueName: \"kubernetes.io/projected/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-kube-api-access-66rsr\") pod \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") "
Oct 30 19:45:04.074631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.074390    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-openshift-service-ca\") pod \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\" (UID: \"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4\") "
Oct 30 19:45:04.074928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.074901    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc642a52-d707-409a-9529-a0e843a06c66-plugins-config-volume" pod "fc642a52-d707-409a-9529-a0e843a06c66" (UID: "fc642a52-d707-409a-9529-a0e843a06c66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:04.075840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.075217    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mh42\" (UniqueName: \"kubernetes.io/projected/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-kube-api-access-6mh42\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4\" (UID: \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:04.075840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.075269    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3359f-38ad-431c-9d07-cab499ec4598-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm\" (UID: \"6ec3359f-38ad-431c-9d07-cab499ec4598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:04.075840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.075399    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4fth\" (UniqueName: \"kubernetes.io/projected/6ec3359f-38ad-431c-9d07-cab499ec4598-kube-api-access-q4fth\") pod \"maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm\" (UID: \"6ec3359f-38ad-431c-9d07-cab499ec4598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:04.075840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.075508    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37631bd2-5738-47fe-93d2-df713a302ff2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85\" (UID: \"37631bd2-5738-47fe-93d2-df713a302ff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:04.076126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.075844    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftmdc\" (UniqueName: \"kubernetes.io/projected/37631bd2-5738-47fe-93d2-df713a302ff2-kube-api-access-ftmdc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85\" (UID: \"37631bd2-5738-47fe-93d2-df713a302ff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:04.076126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.075894    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4\" (UID: \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:04.076126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.075959    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcb5h\" (UniqueName: \"kubernetes.io/projected/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-kube-api-access-kcb5h\") pod \"maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9\" (UID: \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:04.076126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.076016    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-openshift-service-ca" pod "a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" (UID: "a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:04.076339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.076135    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9\" (UID: \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:04.076339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.076270    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8c1a8-a99c-401a-9f79-9657922c2319-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff\" (UID: \"e3d8c1a8-a99c-401a-9f79-9657922c2319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:04.076441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.076345    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/e3d8c1a8-a99c-401a-9f79-9657922c2319-kube-api-access-w8h4h\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff\" (UID: \"e3d8c1a8-a99c-401a-9f79-9657922c2319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:04.076495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.076436    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-metrics-certs" pod "a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" (UID: "a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:04.076729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.076653    6777 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-openshift-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.076729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.076678    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc642a52-d707-409a-9529-a0e843a06c66-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.077690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.077640    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-maas-api-tls" pod "a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" (UID: "a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:04.078584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.078531    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc642a52-d707-409a-9529-a0e843a06c66-kube-api-access-sdjn2" pod "fc642a52-d707-409a-9529-a0e843a06c66" (UID: "fc642a52-d707-409a-9529-a0e843a06c66"). InnerVolumeSpecName "kube-api-access-sdjn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:04.078794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.078720    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"]
Oct 30 19:45:04.079010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.078976    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-kube-api-access-66rsr" pod "a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" (UID: "a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4"). InnerVolumeSpecName "kube-api-access-66rsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:04.083353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.083334    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"]
Oct 30 19:45:04.091004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.090955    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"]
Oct 30 19:45:04.101513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.101481    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a49217-f6xn9"]
Oct 30 19:45:04.120037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.120006    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84556a7c-211f-48e3-9211-ebc8bc5b79d5" path="/var/lib/kubelet/pods/84556a7c-211f-48e3-9211-ebc8bc5b79d5/volumes"
Oct 30 19:45:04.166615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.166564    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"]
Oct 30 19:45:04.171930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.171893    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.175186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.175153    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" event={"ID":"694b1588-e272-4115-ae41-908654b72649","Type":"ContainerStarted","Data":"d6a1dc7512f901d74dc779a7efa2908a9747f1c4c3630ad4c523e34541bb2fd9"}
Oct 30 19:45:04.177893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.177855    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftmdc\" (UniqueName: \"kubernetes.io/projected/37631bd2-5738-47fe-93d2-df713a302ff2-kube-api-access-ftmdc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85\" (UID: \"37631bd2-5738-47fe-93d2-df713a302ff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:04.178043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.177905    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4\" (UID: \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:04.178043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.177951    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcb5h\" (UniqueName: \"kubernetes.io/projected/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-kube-api-access-kcb5h\") pod \"maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9\" (UID: \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:04.178244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178038    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9\" (UID: \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:04.178244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178162    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8c1a8-a99c-401a-9f79-9657922c2319-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff\" (UID: \"e3d8c1a8-a99c-401a-9f79-9657922c2319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:04.178244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178231    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/e3d8c1a8-a99c-401a-9f79-9657922c2319-kube-api-access-w8h4h\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff\" (UID: \"e3d8c1a8-a99c-401a-9f79-9657922c2319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:04.178406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178306    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mh42\" (UniqueName: \"kubernetes.io/projected/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-kube-api-access-6mh42\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4\" (UID: \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:04.178406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178341    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3359f-38ad-431c-9d07-cab499ec4598-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm\" (UID: \"6ec3359f-38ad-431c-9d07-cab499ec4598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:04.178509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178447    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4fth\" (UniqueName: \"kubernetes.io/projected/6ec3359f-38ad-431c-9d07-cab499ec4598-kube-api-access-q4fth\") pod \"maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm\" (UID: \"6ec3359f-38ad-431c-9d07-cab499ec4598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:04.178569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178549    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37631bd2-5738-47fe-93d2-df713a302ff2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85\" (UID: \"37631bd2-5738-47fe-93d2-df713a302ff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:04.178823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178653    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66rsr\" (UniqueName: \"kubernetes.io/projected/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-kube-api-access-66rsr\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.178823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178676    6777 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-metrics-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.178823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178690    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdjn2\" (UniqueName: \"kubernetes.io/projected/fc642a52-d707-409a-9529-a0e843a06c66-kube-api-access-sdjn2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.178823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.178704    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4-maas-api-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.179323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.179297    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9\" (UID: \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:04.179887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.179863    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37631bd2-5738-47fe-93d2-df713a302ff2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85\" (UID: \"37631bd2-5738-47fe-93d2-df713a302ff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:04.180484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.180457    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4\" (UID: \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:04.180608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.180592    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3359f-38ad-431c-9d07-cab499ec4598-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm\" (UID: \"6ec3359f-38ad-431c-9d07-cab499ec4598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:04.180690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.180626    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" event={"ID":"ab87e647-dc82-4dbf-b754-45e414878aeb","Type":"ContainerStarted","Data":"c010436a97dd80e85b4a2c591e280a85f487c8b47bf7e3bd4251ec22515ee03f"}
Oct 30 19:45:04.182036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.182008    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8c1a8-a99c-401a-9f79-9657922c2319-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff\" (UID: \"e3d8c1a8-a99c-401a-9f79-9657922c2319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:04.183544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.183518    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" event={"ID":"6fe76d57-03af-4460-b0ca-84ab80ef060b","Type":"ContainerStarted","Data":"fa0abae97b0889ff9dfa1c1020fdcd5c02126c78a703c23125c8416d6a4b989b"}
Oct 30 19:45:04.186973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.186700    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-826518-openshift-default-dockercfg-bvqnh\""
Oct 30 19:45:04.194751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.194715    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" event={"ID":"debc150f-bdfc-401e-876f-f110044f94b4","Type":"ContainerStarted","Data":"99bc7a923b75c37c71ce1f8f3ef148804b11cc136ace28c585c4a21490f45bec"}
Oct 30 19:45:04.195903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.195753    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"]
Oct 30 19:45:04.197451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.197421    6777 generic.go:356] "Generic (PLEG): container finished" podID="a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" containerID="e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e" exitCode=0
Oct 30 19:45:04.198147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.197558    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx" event={"ID":"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4","Type":"ContainerDied","Data":"e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e"}
Oct 30 19:45:04.198147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.197612    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx" event={"ID":"a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4","Type":"ContainerDied","Data":"1b6e435113159a6528480b8999e758a69d887db7f5627b992a47be9be7643dc7"}
Oct 30 19:45:04.198147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.197654    6777 scope.go:122] "RemoveContainer" containerID="e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e"
Oct 30 19:45:04.198147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.197981    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"
Oct 30 19:45:04.212372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.203675    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/e3d8c1a8-a99c-401a-9f79-9657922c2319-kube-api-access-w8h4h\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff\" (UID: \"e3d8c1a8-a99c-401a-9f79-9657922c2319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:04.212372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.211809    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcb5h\" (UniqueName: \"kubernetes.io/projected/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-kube-api-access-kcb5h\") pod \"maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9\" (UID: \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:04.212372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.211825    6777 generic.go:356] "Generic (PLEG): container finished" podID="fc642a52-d707-409a-9529-a0e843a06c66" containerID="d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5" exitCode=0
Oct 30 19:45:04.212372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.211954    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"
Oct 30 19:45:04.212372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.212020    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss" event={"ID":"fc642a52-d707-409a-9529-a0e843a06c66","Type":"ContainerDied","Data":"d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5"}
Oct 30 19:45:04.212372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.212104    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss" event={"ID":"fc642a52-d707-409a-9529-a0e843a06c66","Type":"ContainerDied","Data":"ec41b55ceea66702a20064cb4048e5b814e15ed3be912eb4ed5db6dc145def3d"}
Oct 30 19:45:04.213445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.212703    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftmdc\" (UniqueName: \"kubernetes.io/projected/37631bd2-5738-47fe-93d2-df713a302ff2-kube-api-access-ftmdc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85\" (UID: \"37631bd2-5738-47fe-93d2-df713a302ff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:04.224594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.224511    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4fth\" (UniqueName: \"kubernetes.io/projected/6ec3359f-38ad-431c-9d07-cab499ec4598-kube-api-access-q4fth\") pod \"maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm\" (UID: \"6ec3359f-38ad-431c-9d07-cab499ec4598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:04.224594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.224576    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mh42\" (UniqueName: \"kubernetes.io/projected/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-kube-api-access-6mh42\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4\" (UID: \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:04.267045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.267017    6777 scope.go:122] "RemoveContainer" containerID="e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e"
Oct 30 19:45:04.267476 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:04.267451    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e\": container with ID starting with e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e not found: ID does not exist" containerID="e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e"
Oct 30 19:45:04.267571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.267485    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e"} err="failed to get container status \"e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e\": rpc error: code = NotFound desc = could not find container \"e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e\": container with ID starting with e48505b5343f3438760bd1fcda0b8eef807a1b4a3d1d0290643c046b612a5e2e not found: ID does not exist"
Oct 30 19:45:04.267571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.267505    6777 scope.go:122] "RemoveContainer" containerID="d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5"
Oct 30 19:45:04.280354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.280316    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-podinfo\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.280797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.280767    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-socket\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.281683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.281655    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-certs\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.281806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.281778    6777 scope.go:122] "RemoveContainer" containerID="d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5"
Oct 30 19:45:04.281806 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.281792    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-credential-socket\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.281918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.281855    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58823786-a7bd-4cc8-9a20-da28e6921be2-istiod-ca-cert\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.282020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.281998    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2mq\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-kube-api-access-kn2mq\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.282152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.282136    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-envoy\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.282198 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:04.282169    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5\": container with ID starting with d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5 not found: ID does not exist" containerID="d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5"
Oct 30 19:45:04.282198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.282182    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-token\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.282273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.282196    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5"} err="failed to get container status \"d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5\": rpc error: code = NotFound desc = could not find container \"d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5\": container with ID starting with d74f57ef93a272501a52f01d0bb07fb2488745cc7e758dcfe8dc02646b2b0dd5 not found: ID does not exist"
Oct 30 19:45:04.282273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.282260    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-data\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.324903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.324864    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"]
Oct 30 19:45:04.332720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.332684    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e6199-77447465d-9dncx"]
Oct 30 19:45:04.340455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.340425    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"]
Oct 30 19:45:04.343401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.343321    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:04.348761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.348736    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-7f4dc89bd7-z65ss"]
Oct 30 19:45:04.384242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384204    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-socket\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.384420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384257    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-certs\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.384420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384352    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-credential-socket\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.384420 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384378    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58823786-a7bd-4cc8-9a20-da28e6921be2-istiod-ca-cert\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.384541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384448    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2mq\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-kube-api-access-kn2mq\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.384541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384536    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-envoy\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.384622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384595    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-token\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.384663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384652    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-data\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.384822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384804    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-podinfo\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.384972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.384809    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-socket\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.385500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.385473    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-certs\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.385640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.385603    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-credential-socket\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.385912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.385886    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-data\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.386002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.385943    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58823786-a7bd-4cc8-9a20-da28e6921be2-istiod-ca-cert\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.387809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.387783    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-envoy\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.388497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.388477    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-podinfo\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.400637 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.400604    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-token\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.401336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.401305    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2mq\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-kube-api-access-kn2mq\") pod \"e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") " pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.415943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.415913    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:04.425481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.425259    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:04.434358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.434052    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:04.447308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.447272    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:04.512214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.512181    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:04.653537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.651861    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"]
Oct 30 19:45:04.660427 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:04.660142    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec3359f_38ad_431c_9d07_cab499ec4598.slice/crio-89b7be6d0c76003a64c9e168b49ced7dacb6c2e4e5fec71b375059e12b1ad658 WatchSource:0}: Error finding container 89b7be6d0c76003a64c9e168b49ced7dacb6c2e4e5fec71b375059e12b1ad658: Status 404 returned error can't find the container with id 89b7be6d0c76003a64c9e168b49ced7dacb6c2e4e5fec71b375059e12b1ad658
Oct 30 19:45:04.723020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.722989    6777 scope.go:122] "RemoveContainer" containerID="42507deb2eca97efa0250bc4477959e861dc7f38f235add9981e4f99f2041cf9"
Oct 30 19:45:04.733158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.733129    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"]
Oct 30 19:45:04.735739 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:04.735706    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37631bd2_5738_47fe_93d2_df713a302ff2.slice/crio-6644f02d29a06209788515f18e2f42843895ef8e040767b555f9881fd8cdbed7 WatchSource:0}: Error finding container 6644f02d29a06209788515f18e2f42843895ef8e040767b555f9881fd8cdbed7: Status 404 returned error can't find the container with id 6644f02d29a06209788515f18e2f42843895ef8e040767b555f9881fd8cdbed7
Oct 30 19:45:04.745861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.745831    6777 scope.go:122] "RemoveContainer" containerID="8392bb7d25def63db052cd228f39dfb832c408f99f694ba30b4fda055c752948"
Oct 30 19:45:04.764242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.764214    6777 scope.go:122] "RemoveContainer" containerID="71b3962a91cef926b9b8891dd84c0d690e75fe438e86e4bad1e5a15569f00c6e"
Oct 30 19:45:04.776070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.776029    6777 scope.go:122] "RemoveContainer" containerID="4b1dc63c99f95b31992598839f4ab3be140e439dc9ed000a1a253c6f53383074"
Oct 30 19:45:04.814635 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.814605    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"]
Oct 30 19:45:04.851771 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.851741    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"]
Oct 30 19:45:04.858105 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:04.858037    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c7be5e_5f69_4a56_b45e_64ded354d9c2.slice/crio-32c41691c1a377719ed43c8215439299ea2956edf1914f33f7de9a836f45550e WatchSource:0}: Error finding container 32c41691c1a377719ed43c8215439299ea2956edf1914f33f7de9a836f45550e: Status 404 returned error can't find the container with id 32c41691c1a377719ed43c8215439299ea2956edf1914f33f7de9a836f45550e
Oct 30 19:45:04.858261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.858149    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"]
Oct 30 19:45:04.869716 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:04.869691    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d8c1a8_a99c_401a_9f79_9657922c2319.slice/crio-a0bcc1bc7865d7a8c4eb0abbcdb6a353274ac3456abfacedde8a9ac4cca280b3 WatchSource:0}: Error finding container a0bcc1bc7865d7a8c4eb0abbcdb6a353274ac3456abfacedde8a9ac4cca280b3: Status 404 returned error can't find the container with id a0bcc1bc7865d7a8c4eb0abbcdb6a353274ac3456abfacedde8a9ac4cca280b3
Oct 30 19:45:04.889361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:04.889336    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"]
Oct 30 19:45:05.052609 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:05.052573    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58823786_a7bd_4cc8_9a20_da28e6921be2.slice/crio-856df76a20383de9072649b091932177053822b66ca02ab568c0f2b9aca49ad2 WatchSource:0}: Error finding container 856df76a20383de9072649b091932177053822b66ca02ab568c0f2b9aca49ad2: Status 404 returned error can't find the container with id 856df76a20383de9072649b091932177053822b66ca02ab568c0f2b9aca49ad2
Oct 30 19:45:05.058076 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.058031    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:05.058170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.058152    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:05.058218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.058208    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:05.223367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.223328    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" event={"ID":"58823786-a7bd-4cc8-9a20-da28e6921be2","Type":"ContainerStarted","Data":"856df76a20383de9072649b091932177053822b66ca02ab568c0f2b9aca49ad2"}
Oct 30 19:45:05.225338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.225307    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" event={"ID":"37631bd2-5738-47fe-93d2-df713a302ff2","Type":"ContainerStarted","Data":"6644f02d29a06209788515f18e2f42843895ef8e040767b555f9881fd8cdbed7"}
Oct 30 19:45:05.258380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.258336    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" event={"ID":"6ec3359f-38ad-431c-9d07-cab499ec4598","Type":"ContainerStarted","Data":"d17e1e88817313d930531eba95858efdbe3c99824003a4f711d54cd1bf0f2df1"}
Oct 30 19:45:05.258646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.258627    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" event={"ID":"6ec3359f-38ad-431c-9d07-cab499ec4598","Type":"ContainerStarted","Data":"89b7be6d0c76003a64c9e168b49ced7dacb6c2e4e5fec71b375059e12b1ad658"}
Oct 30 19:45:05.292775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.292685    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" podStartSLOduration=5.292664629 podStartE2EDuration="5.292664629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.290206003 +0000 UTC m=+2287.287121387" watchObservedRunningTime="2026-10-30 19:45:05.292664629 +0000 UTC m=+2287.289580014"
Oct 30 19:45:05.320488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.320133    6777 generic.go:356] "Generic (PLEG): container finished" podID="694b1588-e272-4115-ae41-908654b72649" containerID="d6a1dc7512f901d74dc779a7efa2908a9747f1c4c3630ad4c523e34541bb2fd9" exitCode=60
Oct 30 19:45:05.320488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.320430    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" event={"ID":"694b1588-e272-4115-ae41-908654b72649","Type":"ContainerDied","Data":"d6a1dc7512f901d74dc779a7efa2908a9747f1c4c3630ad4c523e34541bb2fd9"}
Oct 30 19:45:05.320739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.320495    6777 scope.go:122] "RemoveContainer" containerID="71028a4a98074b39217c7012f2f8665861594c48c8b737a2281147fcfa5e7111"
Oct 30 19:45:05.324808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.324487    6777 scope.go:122] "RemoveContainer" containerID="d6a1dc7512f901d74dc779a7efa2908a9747f1c4c3630ad4c523e34541bb2fd9"
Oct 30 19:45:05.325007 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:05.324983    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-8t28c_odh-ai-gateway-infra(694b1588-e272-4115-ae41-908654b72649)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" podUID="694b1588-e272-4115-ae41-908654b72649"
Oct 30 19:45:05.326402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.326376    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" event={"ID":"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37","Type":"ContainerStarted","Data":"57cf8c93d0cc394e3b7d9abd1f5bf3d1d17002fdb5c033f2e1a5f368e4a866d8"}
Oct 30 19:45:05.330226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.330197    6777 generic.go:356] "Generic (PLEG): container finished" podID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerID="c010436a97dd80e85b4a2c591e280a85f487c8b47bf7e3bd4251ec22515ee03f" exitCode=60
Oct 30 19:45:05.330346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.330324    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" event={"ID":"ab87e647-dc82-4dbf-b754-45e414878aeb","Type":"ContainerDied","Data":"c010436a97dd80e85b4a2c591e280a85f487c8b47bf7e3bd4251ec22515ee03f"}
Oct 30 19:45:05.330644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.330626    6777 scope.go:122] "RemoveContainer" containerID="c010436a97dd80e85b4a2c591e280a85f487c8b47bf7e3bd4251ec22515ee03f"
Oct 30 19:45:05.331113 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:05.331054    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2_odh-ai-gateway-infra(ab87e647-dc82-4dbf-b754-45e414878aeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb"
Oct 30 19:45:05.333305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.333280    6777 generic.go:356] "Generic (PLEG): container finished" podID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerID="fa0abae97b0889ff9dfa1c1020fdcd5c02126c78a703c23125c8416d6a4b989b" exitCode=60
Oct 30 19:45:05.333416 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.333368    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" event={"ID":"6fe76d57-03af-4460-b0ca-84ab80ef060b","Type":"ContainerDied","Data":"fa0abae97b0889ff9dfa1c1020fdcd5c02126c78a703c23125c8416d6a4b989b"}
Oct 30 19:45:05.333801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.333782    6777 scope.go:122] "RemoveContainer" containerID="fa0abae97b0889ff9dfa1c1020fdcd5c02126c78a703c23125c8416d6a4b989b"
Oct 30 19:45:05.334262 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:05.334210    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc_odh-ai-gateway-infra(6fe76d57-03af-4460-b0ca-84ab80ef060b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b"
Oct 30 19:45:05.335931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.335911    6777 generic.go:356] "Generic (PLEG): container finished" podID="debc150f-bdfc-401e-876f-f110044f94b4" containerID="99bc7a923b75c37c71ce1f8f3ef148804b11cc136ace28c585c4a21490f45bec" exitCode=60
Oct 30 19:45:05.336036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.335995    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" event={"ID":"debc150f-bdfc-401e-876f-f110044f94b4","Type":"ContainerDied","Data":"99bc7a923b75c37c71ce1f8f3ef148804b11cc136ace28c585c4a21490f45bec"}
Oct 30 19:45:05.336410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.336389    6777 scope.go:122] "RemoveContainer" containerID="99bc7a923b75c37c71ce1f8f3ef148804b11cc136ace28c585c4a21490f45bec"
Oct 30 19:45:05.336968 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:05.336947    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z_odh-ai-gateway-infra(debc150f-bdfc-401e-876f-f110044f94b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" podUID="debc150f-bdfc-401e-876f-f110044f94b4"
Oct 30 19:45:05.337619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.337596    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" event={"ID":"e3d8c1a8-a99c-401a-9f79-9657922c2319","Type":"ContainerStarted","Data":"a0bcc1bc7865d7a8c4eb0abbcdb6a353274ac3456abfacedde8a9ac4cca280b3"}
Oct 30 19:45:05.339310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.339282    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" event={"ID":"c9c7be5e-5f69-4a56-b45e-64ded354d9c2","Type":"ContainerStarted","Data":"32c41691c1a377719ed43c8215439299ea2956edf1914f33f7de9a836f45550e"}
Oct 30 19:45:05.343215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.343193    6777 scope.go:122] "RemoveContainer" containerID="f8d987a94a6ea9e5b067fecc3b5d6305615495b58eeac850eca520a61c6c12d8"
Oct 30 19:45:05.367849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.367821    6777 scope.go:122] "RemoveContainer" containerID="0aeb966ffc886bdf9533e6038b1faee8b5c30c837aae7bf2b8ac30b9021fd181"
Oct 30 19:45:05.383670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.383651    6777 scope.go:122] "RemoveContainer" containerID="51951878c936cf751a004fc4edd5abc767dfc33d21ad1a57c2fd096e044023d3"
Oct 30 19:45:05.975946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.975901    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:45:05.976202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:05.975997    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:45:06.116001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.115964    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4" path="/var/lib/kubelet/pods/a3aff27e-b7d6-4e22-b0bb-80b8b25e26d4/volumes"
Oct 30 19:45:06.116712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.116687    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc642a52-d707-409a-9529-a0e843a06c66" path="/var/lib/kubelet/pods/fc642a52-d707-409a-9529-a0e843a06c66/volumes"
Oct 30 19:45:06.359240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.359192    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" event={"ID":"c9c7be5e-5f69-4a56-b45e-64ded354d9c2","Type":"ContainerStarted","Data":"2a164bfdd6ad7e3b0680c2b0738880002920552b9ce23c4d70e9f1d7370162ed"}
Oct 30 19:45:06.377946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.367779    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" event={"ID":"58823786-a7bd-4cc8-9a20-da28e6921be2","Type":"ContainerStarted","Data":"87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487"}
Oct 30 19:45:06.377946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.372147    6777 generic.go:356] "Generic (PLEG): container finished" podID="37631bd2-5738-47fe-93d2-df713a302ff2" containerID="7f7134c15807c702e3a53ab3f38ac2a1fa35a33385ed9d8aa35070284d5535dc" exitCode=60
Oct 30 19:45:06.377946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.372256    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" event={"ID":"37631bd2-5738-47fe-93d2-df713a302ff2","Type":"ContainerDied","Data":"7f7134c15807c702e3a53ab3f38ac2a1fa35a33385ed9d8aa35070284d5535dc"}
Oct 30 19:45:06.377946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.372618    6777 scope.go:122] "RemoveContainer" containerID="7f7134c15807c702e3a53ab3f38ac2a1fa35a33385ed9d8aa35070284d5535dc"
Oct 30 19:45:06.377946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.375039    6777 generic.go:356] "Generic (PLEG): container finished" podID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerID="d17e1e88817313d930531eba95858efdbe3c99824003a4f711d54cd1bf0f2df1" exitCode=60
Oct 30 19:45:06.377946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.375211    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" event={"ID":"6ec3359f-38ad-431c-9d07-cab499ec4598","Type":"ContainerDied","Data":"d17e1e88817313d930531eba95858efdbe3c99824003a4f711d54cd1bf0f2df1"}
Oct 30 19:45:06.377946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.376762    6777 scope.go:122] "RemoveContainer" containerID="d17e1e88817313d930531eba95858efdbe3c99824003a4f711d54cd1bf0f2df1"
Oct 30 19:45:06.383376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.383264    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" podStartSLOduration=6.383245736 podStartE2EDuration="6.383245736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:06.382787259 +0000 UTC m=+2288.379702645" watchObservedRunningTime="2026-10-30 19:45:06.383245736 +0000 UTC m=+2288.380161123"
Oct 30 19:45:06.385934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.385908    6777 scope.go:122] "RemoveContainer" containerID="d6a1dc7512f901d74dc779a7efa2908a9747f1c4c3630ad4c523e34541bb2fd9"
Oct 30 19:45:06.386436 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:06.386406    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-8t28c_odh-ai-gateway-infra(694b1588-e272-4115-ae41-908654b72649)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" podUID="694b1588-e272-4115-ae41-908654b72649"
Oct 30 19:45:06.393747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.393720    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" event={"ID":"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37","Type":"ContainerStarted","Data":"fd9898baba0dfaf77034ddfade377e770bc52dc5f0ea8ad3cef341020b01e698"}
Oct 30 19:45:06.394588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.394562    6777 scope.go:122] "RemoveContainer" containerID="fd9898baba0dfaf77034ddfade377e770bc52dc5f0ea8ad3cef341020b01e698"
Oct 30 19:45:06.398977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.398938    6777 scope.go:122] "RemoveContainer" containerID="c010436a97dd80e85b4a2c591e280a85f487c8b47bf7e3bd4251ec22515ee03f"
Oct 30 19:45:06.402482 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:06.402452    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2_odh-ai-gateway-infra(ab87e647-dc82-4dbf-b754-45e414878aeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb"
Oct 30 19:45:06.412326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.412297    6777 scope.go:122] "RemoveContainer" containerID="fa0abae97b0889ff9dfa1c1020fdcd5c02126c78a703c23125c8416d6a4b989b"
Oct 30 19:45:06.412842 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:06.412809    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc_odh-ai-gateway-infra(6fe76d57-03af-4460-b0ca-84ab80ef060b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b"
Oct 30 19:45:06.420462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.420365    6777 scope.go:122] "RemoveContainer" containerID="99bc7a923b75c37c71ce1f8f3ef148804b11cc136ace28c585c4a21490f45bec"
Oct 30 19:45:06.420811 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:06.420786    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z_odh-ai-gateway-infra(debc150f-bdfc-401e-876f-f110044f94b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" podUID="debc150f-bdfc-401e-876f-f110044f94b4"
Oct 30 19:45:06.437906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.437731    6777 generic.go:356] "Generic (PLEG): container finished" podID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerID="656cc1c99c94f2dd9b5d463822322d0d9835e499fe5b6ab912d838adb6c73650" exitCode=60
Oct 30 19:45:06.437906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.437850    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" event={"ID":"e3d8c1a8-a99c-401a-9f79-9657922c2319","Type":"ContainerDied","Data":"656cc1c99c94f2dd9b5d463822322d0d9835e499fe5b6ab912d838adb6c73650"}
Oct 30 19:45:06.438199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.438180    6777 scope.go:122] "RemoveContainer" containerID="656cc1c99c94f2dd9b5d463822322d0d9835e499fe5b6ab912d838adb6c73650"
Oct 30 19:45:06.446217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.446159    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" podStartSLOduration=62.446136972 podStartE2EDuration="1m2.446136972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:06.442054251 +0000 UTC m=+2288.438969643" watchObservedRunningTime="2026-10-30 19:45:06.446136972 +0000 UTC m=+2288.443052357"
Oct 30 19:45:06.512625 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.512584    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:06.518973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:06.518937    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:07.201918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.201829    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"]
Oct 30 19:45:07.202331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.202023    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy" containerID="cri-o://00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f" gracePeriod=30
Oct 30 19:45:07.449124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.449079    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" event={"ID":"e3d8c1a8-a99c-401a-9f79-9657922c2319","Type":"ContainerStarted","Data":"f65424ec0f42803044e15660b6ea9457d4ff4fda1ba98cddc6062758f624e8db"}
Oct 30 19:45:07.452052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.451953    6777 generic.go:356] "Generic (PLEG): container finished" podID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerID="2a164bfdd6ad7e3b0680c2b0738880002920552b9ce23c4d70e9f1d7370162ed" exitCode=60
Oct 30 19:45:07.452052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.452039    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" event={"ID":"c9c7be5e-5f69-4a56-b45e-64ded354d9c2","Type":"ContainerDied","Data":"2a164bfdd6ad7e3b0680c2b0738880002920552b9ce23c4d70e9f1d7370162ed"}
Oct 30 19:45:07.452438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.452418    6777 scope.go:122] "RemoveContainer" containerID="2a164bfdd6ad7e3b0680c2b0738880002920552b9ce23c4d70e9f1d7370162ed"
Oct 30 19:45:07.455119 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.455091    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" event={"ID":"37631bd2-5738-47fe-93d2-df713a302ff2","Type":"ContainerStarted","Data":"b4f6dd2afc5db1bc95bb0dc506c67afff19d71190c7e6178f170c89c5a2a0fce"}
Oct 30 19:45:07.457377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.457353    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" event={"ID":"6ec3359f-38ad-431c-9d07-cab499ec4598","Type":"ContainerStarted","Data":"bd72ac9ed3eaa55f4560413e46aab15d07df335dcd889b6e3a525a5b95d8c599"}
Oct 30 19:45:07.459655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.459631    6777 generic.go:356] "Generic (PLEG): container finished" podID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerID="fd9898baba0dfaf77034ddfade377e770bc52dc5f0ea8ad3cef341020b01e698" exitCode=60
Oct 30 19:45:07.459768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.459703    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" event={"ID":"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37","Type":"ContainerDied","Data":"fd9898baba0dfaf77034ddfade377e770bc52dc5f0ea8ad3cef341020b01e698"}
Oct 30 19:45:07.459768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.459752    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" event={"ID":"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37","Type":"ContainerStarted","Data":"7e4250d4809158356eebe2ac99898214e75f51b2826e09668dc9f8b8a65d56ab"}
Oct 30 19:45:07.460051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.460038    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:07.461204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.461175    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:45:07.501388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.501336    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" podStartSLOduration=7.501319615 podStartE2EDuration="7.501319615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:07.496675251 +0000 UTC m=+2289.493590634" watchObservedRunningTime="2026-10-30 19:45:07.501319615 +0000 UTC m=+2289.498234998"
Oct 30 19:45:07.627384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.627312    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" podStartSLOduration=7.627294119 podStartE2EDuration="7.627294119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:07.625921208 +0000 UTC m=+2289.622836594" watchObservedRunningTime="2026-10-30 19:45:07.627294119 +0000 UTC m=+2289.624209505"
Oct 30 19:45:07.649801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:07.648011    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" podStartSLOduration=7.647987683 podStartE2EDuration="7.647987683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:07.646650733 +0000 UTC m=+2289.643566117" watchObservedRunningTime="2026-10-30 19:45:07.647987683 +0000 UTC m=+2289.644903108"
Oct 30 19:45:08.473886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.473836    6777 generic.go:356] "Generic (PLEG): container finished" podID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerID="f65424ec0f42803044e15660b6ea9457d4ff4fda1ba98cddc6062758f624e8db" exitCode=60
Oct 30 19:45:08.474532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.474100    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" event={"ID":"e3d8c1a8-a99c-401a-9f79-9657922c2319","Type":"ContainerDied","Data":"f65424ec0f42803044e15660b6ea9457d4ff4fda1ba98cddc6062758f624e8db"}
Oct 30 19:45:08.474532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.474197    6777 scope.go:122] "RemoveContainer" containerID="656cc1c99c94f2dd9b5d463822322d0d9835e499fe5b6ab912d838adb6c73650"
Oct 30 19:45:08.474730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.474708    6777 scope.go:122] "RemoveContainer" containerID="f65424ec0f42803044e15660b6ea9457d4ff4fda1ba98cddc6062758f624e8db"
Oct 30 19:45:08.475295 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:08.475264    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff_odh-ai-gateway-infra(e3d8c1a8-a99c-401a-9f79-9657922c2319)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319"
Oct 30 19:45:08.477835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.477807    6777 generic.go:356] "Generic (PLEG): container finished" podID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerID="e6958c58414ce6b7c6cb5a040b603f4ca1460a747e28e8e8d72b821afb6dad45" exitCode=0
Oct 30 19:45:08.477994 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.477891    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" event={"ID":"a4c66346-303b-4d1a-b566-126fa4ea48e3","Type":"ContainerDied","Data":"e6958c58414ce6b7c6cb5a040b603f4ca1460a747e28e8e8d72b821afb6dad45"}
Oct 30 19:45:08.480848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.480819    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" event={"ID":"c9c7be5e-5f69-4a56-b45e-64ded354d9c2","Type":"ContainerStarted","Data":"68776ad904170742bff1e3a9a001398c886995485d717122cea279264440225e"}
Oct 30 19:45:08.484237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.484184    6777 generic.go:356] "Generic (PLEG): container finished" podID="37631bd2-5738-47fe-93d2-df713a302ff2" containerID="b4f6dd2afc5db1bc95bb0dc506c67afff19d71190c7e6178f170c89c5a2a0fce" exitCode=60
Oct 30 19:45:08.484526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.484500    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" event={"ID":"37631bd2-5738-47fe-93d2-df713a302ff2","Type":"ContainerDied","Data":"b4f6dd2afc5db1bc95bb0dc506c67afff19d71190c7e6178f170c89c5a2a0fce"}
Oct 30 19:45:08.484861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.484808    6777 scope.go:122] "RemoveContainer" containerID="b4f6dd2afc5db1bc95bb0dc506c67afff19d71190c7e6178f170c89c5a2a0fce"
Oct 30 19:45:08.485380 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:08.485319    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85_odh-ai-gateway-infra(37631bd2-5738-47fe-93d2-df713a302ff2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" podUID="37631bd2-5738-47fe-93d2-df713a302ff2"
Oct 30 19:45:08.487641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.487614    6777 generic.go:356] "Generic (PLEG): container finished" podID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerID="bd72ac9ed3eaa55f4560413e46aab15d07df335dcd889b6e3a525a5b95d8c599" exitCode=60
Oct 30 19:45:08.487753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.487698    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" event={"ID":"6ec3359f-38ad-431c-9d07-cab499ec4598","Type":"ContainerDied","Data":"bd72ac9ed3eaa55f4560413e46aab15d07df335dcd889b6e3a525a5b95d8c599"}
Oct 30 19:45:08.488511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.488486    6777 scope.go:122] "RemoveContainer" containerID="bd72ac9ed3eaa55f4560413e46aab15d07df335dcd889b6e3a525a5b95d8c599"
Oct 30 19:45:08.488961 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:08.488894    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm_odh-ai-gateway-infra(6ec3359f-38ad-431c-9d07-cab499ec4598)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598"
Oct 30 19:45:08.490041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.490019    6777 generic.go:356] "Generic (PLEG): container finished" podID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerID="7e4250d4809158356eebe2ac99898214e75f51b2826e09668dc9f8b8a65d56ab" exitCode=60
Oct 30 19:45:08.490211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.490150    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" event={"ID":"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37","Type":"ContainerDied","Data":"7e4250d4809158356eebe2ac99898214e75f51b2826e09668dc9f8b8a65d56ab"}
Oct 30 19:45:08.490897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.490877    6777 scope.go:122] "RemoveContainer" containerID="7e4250d4809158356eebe2ac99898214e75f51b2826e09668dc9f8b8a65d56ab"
Oct 30 19:45:08.491551 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:08.491525    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4_odh-ai-gateway-infra(b1a5a0ed-186c-493d-a8c5-ac70ca38fa37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37"
Oct 30 19:45:08.493410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.493391    6777 scope.go:122] "RemoveContainer" containerID="7f7134c15807c702e3a53ab3f38ac2a1fa35a33385ed9d8aa35070284d5535dc"
Oct 30 19:45:08.546670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.546593    6777 scope.go:122] "RemoveContainer" containerID="d17e1e88817313d930531eba95858efdbe3c99824003a4f711d54cd1bf0f2df1"
Oct 30 19:45:08.549884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.549863    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:45:08.561938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.561913    6777 scope.go:122] "RemoveContainer" containerID="fd9898baba0dfaf77034ddfade377e770bc52dc5f0ea8ad3cef341020b01e698"
Oct 30 19:45:08.608945 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.608906    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-token\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-token\") pod \"a4c66346-303b-4d1a-b566-126fa4ea48e3\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") "
Oct 30 19:45:08.609188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.609002    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-podinfo\") pod \"a4c66346-303b-4d1a-b566-126fa4ea48e3\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") "
Oct 30 19:45:08.609188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.609099    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-socket\") pod \"a4c66346-303b-4d1a-b566-126fa4ea48e3\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") "
Oct 30 19:45:08.609188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.609179    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c66346-303b-4d1a-b566-126fa4ea48e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c66346-303b-4d1a-b566-126fa4ea48e3-istiod-ca-cert\") pod \"a4c66346-303b-4d1a-b566-126fa4ea48e3\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") "
Oct 30 19:45:08.609405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.609268    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-envoy\") pod \"a4c66346-303b-4d1a-b566-126fa4ea48e3\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") "
Oct 30 19:45:08.609841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.609660    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-kube-api-access-lsj75\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-kube-api-access-lsj75\") pod \"a4c66346-303b-4d1a-b566-126fa4ea48e3\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") "
Oct 30 19:45:08.609841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.609700    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c66346-303b-4d1a-b566-126fa4ea48e3-istiod-ca-cert" pod "a4c66346-303b-4d1a-b566-126fa4ea48e3" (UID: "a4c66346-303b-4d1a-b566-126fa4ea48e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:08.609841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.609730    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-certs\") pod \"a4c66346-303b-4d1a-b566-126fa4ea48e3\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") "
Oct 30 19:45:08.609841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.609783    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-socket" pod "a4c66346-303b-4d1a-b566-126fa4ea48e3" (UID: "a4c66346-303b-4d1a-b566-126fa4ea48e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:08.610249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.610010    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-data\") pod \"a4c66346-303b-4d1a-b566-126fa4ea48e3\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") "
Oct 30 19:45:08.610249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.610040    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-credential-socket\") pod \"a4c66346-303b-4d1a-b566-126fa4ea48e3\" (UID: \"a4c66346-303b-4d1a-b566-126fa4ea48e3\") "
Oct 30 19:45:08.610249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.610074    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-certs" pod "a4c66346-303b-4d1a-b566-126fa4ea48e3" (UID: "a4c66346-303b-4d1a-b566-126fa4ea48e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:08.610409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.610258    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-data" pod "a4c66346-303b-4d1a-b566-126fa4ea48e3" (UID: "a4c66346-303b-4d1a-b566-126fa4ea48e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:08.610805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.610455    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-credential-socket" pod "a4c66346-303b-4d1a-b566-126fa4ea48e3" (UID: "a4c66346-303b-4d1a-b566-126fa4ea48e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:08.612251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.612225    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:08.612344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.612253    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c66346-303b-4d1a-b566-126fa4ea48e3-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:08.612344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.612267    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:08.612344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.612280    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:08.612344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.612293    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:08.614725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.614704    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"]
Oct 30 19:45:08.615655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.615632    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy"
Oct 30 19:45:08.615655 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.615652    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy"
Oct 30 19:45:08.615844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.615833    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" containerName="istio-proxy"
Oct 30 19:45:08.617024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.616993    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-envoy" pod "a4c66346-303b-4d1a-b566-126fa4ea48e3" (UID: "a4c66346-303b-4d1a-b566-126fa4ea48e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:08.617165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.617100    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-token" pod "a4c66346-303b-4d1a-b566-126fa4ea48e3" (UID: "a4c66346-303b-4d1a-b566-126fa4ea48e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:08.617165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.617109    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-kube-api-access-lsj75" pod "a4c66346-303b-4d1a-b566-126fa4ea48e3" (UID: "a4c66346-303b-4d1a-b566-126fa4ea48e3"). InnerVolumeSpecName "kube-api-access-lsj75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:08.617269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.617162    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-podinfo" pod "a4c66346-303b-4d1a-b566-126fa4ea48e3" (UID: "a4c66346-303b-4d1a-b566-126fa4ea48e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:08.625349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.625320    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.630776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.630610    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"]
Oct 30 19:45:08.632415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.632383    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c2b092-serving-cert\""
Oct 30 19:45:08.717681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.716472    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2ceff9-2813-4c3d-8847-8ddbbab67904-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.717681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.717130    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-metrics-certs\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.717681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.717333    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.717681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.717367    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5kj\" (UniqueName: \"kubernetes.io/projected/ae2ceff9-2813-4c3d-8847-8ddbbab67904-kube-api-access-4h5kj\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.717681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.717551    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:08.717681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.717570    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:08.717681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.717584    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c66346-303b-4d1a-b566-126fa4ea48e3-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:08.717681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.717598    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsj75\" (UniqueName: \"kubernetes.io/projected/a4c66346-303b-4d1a-b566-126fa4ea48e3-kube-api-access-lsj75\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:08.819262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.819229    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-metrics-certs\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.819474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.819410    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.819474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.819442    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5kj\" (UniqueName: \"kubernetes.io/projected/ae2ceff9-2813-4c3d-8847-8ddbbab67904-kube-api-access-4h5kj\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.819597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.819501    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2ceff9-2813-4c3d-8847-8ddbbab67904-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.820211 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.820184    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2ceff9-2813-4c3d-8847-8ddbbab67904-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.821837 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.821813    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-metrics-certs\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.821947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.821842    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:08.832798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:08.832729    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5kj\" (UniqueName: \"kubernetes.io/projected/ae2ceff9-2813-4c3d-8847-8ddbbab67904-kube-api-access-4h5kj\") pod \"maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:09.044599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.044559    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:09.347351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.347321    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"]
Oct 30 19:45:09.685079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.683096    6777 scope.go:122] "RemoveContainer" containerID="b4f6dd2afc5db1bc95bb0dc506c67afff19d71190c7e6178f170c89c5a2a0fce"
Oct 30 19:45:09.685079 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:09.683770    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85_odh-ai-gateway-infra(37631bd2-5738-47fe-93d2-df713a302ff2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" podUID="37631bd2-5738-47fe-93d2-df713a302ff2"
Oct 30 19:45:09.701082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.697332    6777 scope.go:122] "RemoveContainer" containerID="bd72ac9ed3eaa55f4560413e46aab15d07df335dcd889b6e3a525a5b95d8c599"
Oct 30 19:45:09.701082 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:09.697891    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm_odh-ai-gateway-infra(6ec3359f-38ad-431c-9d07-cab499ec4598)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598"
Oct 30 19:45:09.704092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.702210    6777 scope.go:122] "RemoveContainer" containerID="7e4250d4809158356eebe2ac99898214e75f51b2826e09668dc9f8b8a65d56ab"
Oct 30 19:45:09.704092 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:09.702798    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4_odh-ai-gateway-infra(b1a5a0ed-186c-493d-a8c5-ac70ca38fa37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37"
Oct 30 19:45:09.727539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.726917    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf" event={"ID":"ae2ceff9-2813-4c3d-8847-8ddbbab67904","Type":"ContainerStarted","Data":"8a981f770d2f5588a2ee6c501cb59e39ed5d52870f9ffb17adf46313909f610c"}
Oct 30 19:45:09.730887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.730856    6777 scope.go:122] "RemoveContainer" containerID="f65424ec0f42803044e15660b6ea9457d4ff4fda1ba98cddc6062758f624e8db"
Oct 30 19:45:09.740864 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:09.740825    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff_odh-ai-gateway-infra(e3d8c1a8-a99c-401a-9f79-9657922c2319)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319"
Oct 30 19:45:09.750543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.750503    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg" event={"ID":"a4c66346-303b-4d1a-b566-126fa4ea48e3","Type":"ContainerDied","Data":"421f43a6b364418d62526b2613460e362cf07f36fade9c92c235423710e10ea7"}
Oct 30 19:45:09.750724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.750592    6777 scope.go:122] "RemoveContainer" containerID="e6958c58414ce6b7c6cb5a040b603f4ca1460a747e28e8e8d72b821afb6dad45"
Oct 30 19:45:09.750781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.750767    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"
Oct 30 19:45:09.758199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.758164    6777 generic.go:356] "Generic (PLEG): container finished" podID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerID="68776ad904170742bff1e3a9a001398c886995485d717122cea279264440225e" exitCode=60
Oct 30 19:45:09.758345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.758271    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" event={"ID":"c9c7be5e-5f69-4a56-b45e-64ded354d9c2","Type":"ContainerDied","Data":"68776ad904170742bff1e3a9a001398c886995485d717122cea279264440225e"}
Oct 30 19:45:09.758769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.758739    6777 scope.go:122] "RemoveContainer" containerID="68776ad904170742bff1e3a9a001398c886995485d717122cea279264440225e"
Oct 30 19:45:09.759412 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:09.759386    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9_odh-ai-gateway-infra(c9c7be5e-5f69-4a56-b45e-64ded354d9c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2"
Oct 30 19:45:09.770747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.770724    6777 scope.go:122] "RemoveContainer" containerID="2a164bfdd6ad7e3b0680c2b0738880002920552b9ce23c4d70e9f1d7370162ed"
Oct 30 19:45:09.827289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.827254    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"]
Oct 30 19:45:09.838183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:09.838147    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-x8fbg"]
Oct 30 19:45:10.124041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:10.124005    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c66346-303b-4d1a-b566-126fa4ea48e3" path="/var/lib/kubelet/pods/a4c66346-303b-4d1a-b566-126fa4ea48e3/volumes"
Oct 30 19:45:10.819088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:10.816892    6777 scope.go:122] "RemoveContainer" containerID="68776ad904170742bff1e3a9a001398c886995485d717122cea279264440225e"
Oct 30 19:45:10.819088 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:10.817406    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9_odh-ai-gateway-infra(c9c7be5e-5f69-4a56-b45e-64ded354d9c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2"
Oct 30 19:45:10.835105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:10.835048    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf" event={"ID":"ae2ceff9-2813-4c3d-8847-8ddbbab67904","Type":"ContainerStarted","Data":"659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4"}
Oct 30 19:45:10.844127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:10.836581    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:10.892949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:10.892886    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf" podStartSLOduration=5.35122181 podStartE2EDuration="5.892867584s" totalImagesPullingTime="541.645774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:10.892468012 +0000 UTC m=+2292.889383396" watchObservedRunningTime="2026-10-30 19:45:10.892867584 +0000 UTC m=+2292.889782970"
Oct 30 19:45:11.933718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:11.933678    6777 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:45:11.934353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:11.934312    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:45:12.641214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.641179    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:45:12.688851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.688813    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"]
Oct 30 19:45:12.689856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.689835    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy"
Oct 30 19:45:12.689913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.689860    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy"
Oct 30 19:45:12.690093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.690081    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerName="istio-proxy"
Oct 30 19:45:12.694141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.694112    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:12.700775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.698498    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-826518\""
Oct 30 19:45:12.700775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.698698    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-826518-dockercfg-g6hrn\""
Oct 30 19:45:12.702932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.702900    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"]
Oct 30 19:45:12.743711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.743667    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-kube-api-access-6cv6v\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-kube-api-access-6cv6v\") pod \"8fe14650-0968-4dec-9701-ab16d1a8b347\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") "
Oct 30 19:45:12.743969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.743714    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-socket\") pod \"8fe14650-0968-4dec-9701-ab16d1a8b347\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") "
Oct 30 19:45:12.743969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.743801    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-data\") pod \"8fe14650-0968-4dec-9701-ab16d1a8b347\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") "
Oct 30 19:45:12.743969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.743848    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fe14650-0968-4dec-9701-ab16d1a8b347-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe14650-0968-4dec-9701-ab16d1a8b347-istiod-ca-cert\") pod \"8fe14650-0968-4dec-9701-ab16d1a8b347\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") "
Oct 30 19:45:12.743969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.743900    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-envoy\") pod \"8fe14650-0968-4dec-9701-ab16d1a8b347\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") "
Oct 30 19:45:12.744178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.744108    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-socket" pod "8fe14650-0968-4dec-9701-ab16d1a8b347" (UID: "8fe14650-0968-4dec-9701-ab16d1a8b347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.744178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.744156    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-token\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-token\") pod \"8fe14650-0968-4dec-9701-ab16d1a8b347\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") "
Oct 30 19:45:12.744697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.744362    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-podinfo\") pod \"8fe14650-0968-4dec-9701-ab16d1a8b347\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") "
Oct 30 19:45:12.744697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.744446    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-certs\") pod \"8fe14650-0968-4dec-9701-ab16d1a8b347\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") "
Oct 30 19:45:12.744697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.744512    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-credential-socket\") pod \"8fe14650-0968-4dec-9701-ab16d1a8b347\" (UID: \"8fe14650-0968-4dec-9701-ab16d1a8b347\") "
Oct 30 19:45:12.745109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.745053    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklml\" (UniqueName: \"kubernetes.io/projected/bf77670b-ea82-4af3-98a8-c112442539a3-kube-api-access-zklml\") pod \"payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb\" (UID: \"bf77670b-ea82-4af3-98a8-c112442539a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:12.745437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.745407    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf77670b-ea82-4af3-98a8-c112442539a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb\" (UID: \"bf77670b-ea82-4af3-98a8-c112442539a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:12.745543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.745522    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-data" pod "8fe14650-0968-4dec-9701-ab16d1a8b347" (UID: "8fe14650-0968-4dec-9701-ab16d1a8b347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.745608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.745586    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-certs" pod "8fe14650-0968-4dec-9701-ab16d1a8b347" (UID: "8fe14650-0968-4dec-9701-ab16d1a8b347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.745738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.745713    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-credential-socket" pod "8fe14650-0968-4dec-9701-ab16d1a8b347" (UID: "8fe14650-0968-4dec-9701-ab16d1a8b347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.746050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.746024    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.746050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.746052    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.746256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.746113    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.746256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.746127    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.746458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.746435    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe14650-0968-4dec-9701-ab16d1a8b347-istiod-ca-cert" pod "8fe14650-0968-4dec-9701-ab16d1a8b347" (UID: "8fe14650-0968-4dec-9701-ab16d1a8b347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:12.746619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.746593    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-kube-api-access-6cv6v" pod "8fe14650-0968-4dec-9701-ab16d1a8b347" (UID: "8fe14650-0968-4dec-9701-ab16d1a8b347"). InnerVolumeSpecName "kube-api-access-6cv6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:12.746744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.746632    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-token" pod "8fe14650-0968-4dec-9701-ab16d1a8b347" (UID: "8fe14650-0968-4dec-9701-ab16d1a8b347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:12.746744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.746709    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-envoy" pod "8fe14650-0968-4dec-9701-ab16d1a8b347" (UID: "8fe14650-0968-4dec-9701-ab16d1a8b347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.747397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.747371    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-podinfo" pod "8fe14650-0968-4dec-9701-ab16d1a8b347" (UID: "8fe14650-0968-4dec-9701-ab16d1a8b347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:12.848035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.847990    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklml\" (UniqueName: \"kubernetes.io/projected/bf77670b-ea82-4af3-98a8-c112442539a3-kube-api-access-zklml\") pod \"payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb\" (UID: \"bf77670b-ea82-4af3-98a8-c112442539a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:12.848553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.848500    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf77670b-ea82-4af3-98a8-c112442539a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb\" (UID: \"bf77670b-ea82-4af3-98a8-c112442539a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:12.848893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.848866    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.849021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.848894    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.849021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.848913    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cv6v\" (UniqueName: \"kubernetes.io/projected/8fe14650-0968-4dec-9701-ab16d1a8b347-kube-api-access-6cv6v\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.849021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.848928    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe14650-0968-4dec-9701-ab16d1a8b347-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.849021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.848942    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe14650-0968-4dec-9701-ab16d1a8b347-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.849862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.849779    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf77670b-ea82-4af3-98a8-c112442539a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb\" (UID: \"bf77670b-ea82-4af3-98a8-c112442539a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:12.858668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.858641    6777 generic.go:356] "Generic (PLEG): container finished" podID="8fe14650-0968-4dec-9701-ab16d1a8b347" containerID="00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f" exitCode=0
Oct 30 19:45:12.858775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.858702    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"
Oct 30 19:45:12.858775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.858724    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" event={"ID":"8fe14650-0968-4dec-9701-ab16d1a8b347","Type":"ContainerDied","Data":"00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f"}
Oct 30 19:45:12.858910 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.858817    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m" event={"ID":"8fe14650-0968-4dec-9701-ab16d1a8b347","Type":"ContainerDied","Data":"fecf557bac2036d197a86f13f31a2b63ff8ae6f62ee52bae59ac19f1519b84cf"}
Oct 30 19:45:12.858910 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.858852    6777 scope.go:122] "RemoveContainer" containerID="00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f"
Oct 30 19:45:12.860949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.860924    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklml\" (UniqueName: \"kubernetes.io/projected/bf77670b-ea82-4af3-98a8-c112442539a3-kube-api-access-zklml\") pod \"payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb\" (UID: \"bf77670b-ea82-4af3-98a8-c112442539a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:12.876623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.876597    6777 scope.go:122] "RemoveContainer" containerID="00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f"
Oct 30 19:45:12.876963 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:12.876942    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f\": container with ID starting with 00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f not found: ID does not exist" containerID="00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f"
Oct 30 19:45:12.877050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.876977    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f"} err="failed to get container status \"00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f\": rpc error: code = NotFound desc = could not find container \"00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f\": container with ID starting with 00d34efef577472ba54fefeabe8a58694d331eba7fd7a36183668fe8687c472f not found: ID does not exist"
Oct 30 19:45:12.914129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.914084    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"]
Oct 30 19:45:12.918325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:12.918298    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-fx76m"]
Oct 30 19:45:13.017340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:13.017307    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:13.318732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:13.318704    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"]
Oct 30 19:45:13.323521 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:13.323487    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf77670b_ea82_4af3_98a8_c112442539a3.slice/crio-51bdd9a24adbc318bde56a26dc62c04068607b63581e23514cb69e7f632d15e4 WatchSource:0}: Error finding container 51bdd9a24adbc318bde56a26dc62c04068607b63581e23514cb69e7f632d15e4: Status 404 returned error can't find the container with id 51bdd9a24adbc318bde56a26dc62c04068607b63581e23514cb69e7f632d15e4
Oct 30 19:45:13.784769 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:13.784724    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:45:13.873408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:13.873304    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb" event={"ID":"bf77670b-ea82-4af3-98a8-c112442539a3","Type":"ContainerStarted","Data":"b52c60e70c0bae9e275d98eed38249d080ce96a3078b71c90c72ebd93809e182"}
Oct 30 19:45:13.873408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:13.873359    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb" event={"ID":"bf77670b-ea82-4af3-98a8-c112442539a3","Type":"ContainerStarted","Data":"51bdd9a24adbc318bde56a26dc62c04068607b63581e23514cb69e7f632d15e4"}
Oct 30 19:45:13.873408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:13.873389    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:13.904514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:13.904444    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb" podStartSLOduration=3.904427682 podStartE2EDuration="3.904427682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:13.895875553 +0000 UTC m=+2295.892790937" watchObservedRunningTime="2026-10-30 19:45:13.904427682 +0000 UTC m=+2295.901343149"
Oct 30 19:45:14.121801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:14.121768    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe14650-0968-4dec-9701-ab16d1a8b347" path="/var/lib/kubelet/pods/8fe14650-0968-4dec-9701-ab16d1a8b347/volumes"
Oct 30 19:45:16.105563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:16.105531    6777 scope.go:122] "RemoveContainer" containerID="d6a1dc7512f901d74dc779a7efa2908a9747f1c4c3630ad4c523e34541bb2fd9"
Oct 30 19:45:16.916232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:16.916190    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" event={"ID":"694b1588-e272-4115-ae41-908654b72649","Type":"ContainerStarted","Data":"9503e56be9b77d9640f83aa328baaba0c58a89cd3d463f7054d1b811eaed5a22"}
Oct 30 19:45:17.106393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:17.106298    6777 scope.go:122] "RemoveContainer" containerID="fa0abae97b0889ff9dfa1c1020fdcd5c02126c78a703c23125c8416d6a4b989b"
Oct 30 19:45:17.879079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:17.869029    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:45:17.929744 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:17.929707    6777 generic.go:356] "Generic (PLEG): container finished" podID="694b1588-e272-4115-ae41-908654b72649" containerID="9503e56be9b77d9640f83aa328baaba0c58a89cd3d463f7054d1b811eaed5a22" exitCode=60
Oct 30 19:45:17.929969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:17.929787    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" event={"ID":"694b1588-e272-4115-ae41-908654b72649","Type":"ContainerDied","Data":"9503e56be9b77d9640f83aa328baaba0c58a89cd3d463f7054d1b811eaed5a22"}
Oct 30 19:45:17.929969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:17.929850    6777 scope.go:122] "RemoveContainer" containerID="d6a1dc7512f901d74dc779a7efa2908a9747f1c4c3630ad4c523e34541bb2fd9"
Oct 30 19:45:17.930468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:17.930443    6777 scope.go:122] "RemoveContainer" containerID="9503e56be9b77d9640f83aa328baaba0c58a89cd3d463f7054d1b811eaed5a22"
Oct 30 19:45:17.931267 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:17.930966    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-8t28c_odh-ai-gateway-infra(694b1588-e272-4115-ae41-908654b72649)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" podUID="694b1588-e272-4115-ae41-908654b72649"
Oct 30 19:45:17.938045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:17.937958    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" event={"ID":"6fe76d57-03af-4460-b0ca-84ab80ef060b","Type":"ContainerStarted","Data":"143246fb32c79b2af8d9f82c06b3e081bef86768b969d64b2dcc754ce15de3cd"}
Oct 30 19:45:17.965203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:17.965135    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"]
Oct 30 19:45:18.128201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:18.128152    6777 scope.go:122] "RemoveContainer" containerID="c010436a97dd80e85b4a2c591e280a85f487c8b47bf7e3bd4251ec22515ee03f"
Oct 30 19:45:18.128680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:18.128394    6777 scope.go:122] "RemoveContainer" containerID="99bc7a923b75c37c71ce1f8f3ef148804b11cc136ace28c585c4a21490f45bec"
Oct 30 19:45:18.979868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:18.979833    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" event={"ID":"ab87e647-dc82-4dbf-b754-45e414878aeb","Type":"ContainerStarted","Data":"28ae62994c1997a106a37cc1dd04f3672ae87176debb29524010455a974c43b5"}
Oct 30 19:45:18.982264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:18.982228    6777 generic.go:356] "Generic (PLEG): container finished" podID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerID="143246fb32c79b2af8d9f82c06b3e081bef86768b969d64b2dcc754ce15de3cd" exitCode=60
Oct 30 19:45:18.982540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:18.982294    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" event={"ID":"6fe76d57-03af-4460-b0ca-84ab80ef060b","Type":"ContainerDied","Data":"143246fb32c79b2af8d9f82c06b3e081bef86768b969d64b2dcc754ce15de3cd"}
Oct 30 19:45:18.982540 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:18.982370    6777 scope.go:122] "RemoveContainer" containerID="fa0abae97b0889ff9dfa1c1020fdcd5c02126c78a703c23125c8416d6a4b989b"
Oct 30 19:45:18.983417 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:18.983398    6777 scope.go:122] "RemoveContainer" containerID="143246fb32c79b2af8d9f82c06b3e081bef86768b969d64b2dcc754ce15de3cd"
Oct 30 19:45:18.983915 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:18.983871    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc_odh-ai-gateway-infra(6fe76d57-03af-4460-b0ca-84ab80ef060b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b"
Oct 30 19:45:18.985615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:18.985582    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" event={"ID":"debc150f-bdfc-401e-876f-f110044f94b4","Type":"ContainerStarted","Data":"d181af62f7ef15af9cef19e914b308b08ae8e153d42fd0a0ff6aaa79f596296f"}
Oct 30 19:45:19.047838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.047138    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"]
Oct 30 19:45:19.108606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.108571    6777 scope.go:122] "RemoveContainer" containerID="b4f6dd2afc5db1bc95bb0dc506c67afff19d71190c7e6178f170c89c5a2a0fce"
Oct 30 19:45:19.108826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.108807    6777 scope.go:122] "RemoveContainer" containerID="68776ad904170742bff1e3a9a001398c886995485d717122cea279264440225e"
Oct 30 19:45:19.338378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.338349    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:19.490448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.479303    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694b1588-e272-4115-ae41-908654b72649-kube-api-access-gg5k6\" (UniqueName: \"kubernetes.io/projected/694b1588-e272-4115-ae41-908654b72649-kube-api-access-gg5k6\") pod \"694b1588-e272-4115-ae41-908654b72649\" (UID: \"694b1588-e272-4115-ae41-908654b72649\") "
Oct 30 19:45:19.490448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.479417    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694b1588-e272-4115-ae41-908654b72649-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/694b1588-e272-4115-ae41-908654b72649-maas-api-service-ca\") pod \"694b1588-e272-4115-ae41-908654b72649\" (UID: \"694b1588-e272-4115-ae41-908654b72649\") "
Oct 30 19:45:19.492130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.492024    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694b1588-e272-4115-ae41-908654b72649-kube-api-access-gg5k6" pod "694b1588-e272-4115-ae41-908654b72649" (UID: "694b1588-e272-4115-ae41-908654b72649"). InnerVolumeSpecName "kube-api-access-gg5k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.492337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.492283    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694b1588-e272-4115-ae41-908654b72649-maas-api-service-ca" pod "694b1588-e272-4115-ae41-908654b72649" (UID: "694b1588-e272-4115-ae41-908654b72649"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.582455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.582356    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5k6\" (UniqueName: \"kubernetes.io/projected/694b1588-e272-4115-ae41-908654b72649-kube-api-access-gg5k6\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.582455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:19.582394    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/694b1588-e272-4115-ae41-908654b72649-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.001956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.001918    6777 generic.go:356] "Generic (PLEG): container finished" podID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerID="28ae62994c1997a106a37cc1dd04f3672ae87176debb29524010455a974c43b5" exitCode=60
Oct 30 19:45:20.001956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.001945    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" event={"ID":"ab87e647-dc82-4dbf-b754-45e414878aeb","Type":"ContainerDied","Data":"28ae62994c1997a106a37cc1dd04f3672ae87176debb29524010455a974c43b5"}
Oct 30 19:45:20.002252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.002096    6777 scope.go:122] "RemoveContainer" containerID="c010436a97dd80e85b4a2c591e280a85f487c8b47bf7e3bd4251ec22515ee03f"
Oct 30 19:45:20.002606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.002580    6777 scope.go:122] "RemoveContainer" containerID="28ae62994c1997a106a37cc1dd04f3672ae87176debb29524010455a974c43b5"
Oct 30 19:45:20.003072 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:20.003011    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2_odh-ai-gateway-infra(ab87e647-dc82-4dbf-b754-45e414878aeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb"
Oct 30 19:45:20.007526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.007499    6777 generic.go:356] "Generic (PLEG): container finished" podID="debc150f-bdfc-401e-876f-f110044f94b4" containerID="d181af62f7ef15af9cef19e914b308b08ae8e153d42fd0a0ff6aaa79f596296f" exitCode=60
Oct 30 19:45:20.007680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.007660    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" event={"ID":"debc150f-bdfc-401e-876f-f110044f94b4","Type":"ContainerDied","Data":"d181af62f7ef15af9cef19e914b308b08ae8e153d42fd0a0ff6aaa79f596296f"}
Oct 30 19:45:20.008130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.008042    6777 scope.go:122] "RemoveContainer" containerID="d181af62f7ef15af9cef19e914b308b08ae8e153d42fd0a0ff6aaa79f596296f"
Oct 30 19:45:20.008457 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:20.008434    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z_odh-ai-gateway-infra(debc150f-bdfc-401e-876f-f110044f94b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" podUID="debc150f-bdfc-401e-876f-f110044f94b4"
Oct 30 19:45:20.010597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.010565    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" event={"ID":"c9c7be5e-5f69-4a56-b45e-64ded354d9c2","Type":"ContainerStarted","Data":"2c3ff24fc0a03b28c123281d405dc07f844d563422f932ff3ea7b627d0dc40b0"}
Oct 30 19:45:20.013365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.013171    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" event={"ID":"37631bd2-5738-47fe-93d2-df713a302ff2","Type":"ContainerStarted","Data":"73c7e8b8d2bea15f4ad3939dc54c5d2052a842a6bfac127e8ee551ce8374d96f"}
Oct 30 19:45:20.017503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.017473    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c" event={"ID":"694b1588-e272-4115-ae41-908654b72649","Type":"ContainerDied","Data":"2a1d7e7dbab58f3154da0ae24f38324e715dacddf4145a9ed9ea9dd51b8355cd"}
Oct 30 19:45:20.017503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.015902    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"
Oct 30 19:45:20.022150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.022126    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"]
Oct 30 19:45:20.024579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.024541    6777 scope.go:122] "RemoveContainer" containerID="99bc7a923b75c37c71ce1f8f3ef148804b11cc136ace28c585c4a21490f45bec"
Oct 30 19:45:20.085304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.085247    6777 scope.go:122] "RemoveContainer" containerID="9503e56be9b77d9640f83aa328baaba0c58a89cd3d463f7054d1b811eaed5a22"
Oct 30 19:45:20.092293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.092175    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"]
Oct 30 19:45:20.109760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.109727    6777 scope.go:122] "RemoveContainer" containerID="7e4250d4809158356eebe2ac99898214e75f51b2826e09668dc9f8b8a65d56ab"
Oct 30 19:45:20.109920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.109828    6777 scope.go:122] "RemoveContainer" containerID="bd72ac9ed3eaa55f4560413e46aab15d07df335dcd889b6e3a525a5b95d8c599"
Oct 30 19:45:20.109920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.109911    6777 scope.go:122] "RemoveContainer" containerID="f65424ec0f42803044e15660b6ea9457d4ff4fda1ba98cddc6062758f624e8db"
Oct 30 19:45:20.236049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.236020    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"]
Oct 30 19:45:20.242595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.242558    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-8t28c"]
Oct 30 19:45:20.385185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.376317    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:20.538310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.528576    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe76d57-03af-4460-b0ca-84ab80ef060b-kube-api-access-ck74g\" (UniqueName: \"kubernetes.io/projected/6fe76d57-03af-4460-b0ca-84ab80ef060b-kube-api-access-ck74g\") pod \"6fe76d57-03af-4460-b0ca-84ab80ef060b\" (UID: \"6fe76d57-03af-4460-b0ca-84ab80ef060b\") "
Oct 30 19:45:20.538310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.528628    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe76d57-03af-4460-b0ca-84ab80ef060b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe76d57-03af-4460-b0ca-84ab80ef060b-maas-api-service-ca\") pod \"6fe76d57-03af-4460-b0ca-84ab80ef060b\" (UID: \"6fe76d57-03af-4460-b0ca-84ab80ef060b\") "
Oct 30 19:45:20.554095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.551377    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe76d57-03af-4460-b0ca-84ab80ef060b-maas-api-service-ca" pod "6fe76d57-03af-4460-b0ca-84ab80ef060b" (UID: "6fe76d57-03af-4460-b0ca-84ab80ef060b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:20.587895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.586319    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe76d57-03af-4460-b0ca-84ab80ef060b-kube-api-access-ck74g" pod "6fe76d57-03af-4460-b0ca-84ab80ef060b" (UID: "6fe76d57-03af-4460-b0ca-84ab80ef060b"). InnerVolumeSpecName "kube-api-access-ck74g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:20.632944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.632895    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck74g\" (UniqueName: \"kubernetes.io/projected/6fe76d57-03af-4460-b0ca-84ab80ef060b-kube-api-access-ck74g\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.633257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:20.633233    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe76d57-03af-4460-b0ca-84ab80ef060b-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.034834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.034386    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc" event={"ID":"6fe76d57-03af-4460-b0ca-84ab80ef060b","Type":"ContainerDied","Data":"84a492c5c5d214cfab90dd2ba6d5c1130fb07cdc50e524df8648c189663f306d"}
Oct 30 19:45:21.034834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.034437    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"
Oct 30 19:45:21.034834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.034513    6777 scope.go:122] "RemoveContainer" containerID="143246fb32c79b2af8d9f82c06b3e081bef86768b969d64b2dcc754ce15de3cd"
Oct 30 19:45:21.040793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.040760    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" event={"ID":"e3d8c1a8-a99c-401a-9f79-9657922c2319","Type":"ContainerStarted","Data":"7f93bf8a074445ca6a6cc85b3b6e914c3a6054ccc9875a9fcebeb11a80096168"}
Oct 30 19:45:21.044476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.044445    6777 generic.go:356] "Generic (PLEG): container finished" podID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerID="2c3ff24fc0a03b28c123281d405dc07f844d563422f932ff3ea7b627d0dc40b0" exitCode=60
Oct 30 19:45:21.044595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.044477    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" event={"ID":"c9c7be5e-5f69-4a56-b45e-64ded354d9c2","Type":"ContainerDied","Data":"2c3ff24fc0a03b28c123281d405dc07f844d563422f932ff3ea7b627d0dc40b0"}
Oct 30 19:45:21.044936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.044913    6777 scope.go:122] "RemoveContainer" containerID="2c3ff24fc0a03b28c123281d405dc07f844d563422f932ff3ea7b627d0dc40b0"
Oct 30 19:45:21.045371 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:21.045335    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9_odh-ai-gateway-infra(c9c7be5e-5f69-4a56-b45e-64ded354d9c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2"
Oct 30 19:45:21.064527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.048182    6777 generic.go:356] "Generic (PLEG): container finished" podID="37631bd2-5738-47fe-93d2-df713a302ff2" containerID="73c7e8b8d2bea15f4ad3939dc54c5d2052a842a6bfac127e8ee551ce8374d96f" exitCode=60
Oct 30 19:45:21.079390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.052136    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" event={"ID":"37631bd2-5738-47fe-93d2-df713a302ff2","Type":"ContainerDied","Data":"73c7e8b8d2bea15f4ad3939dc54c5d2052a842a6bfac127e8ee551ce8374d96f"}
Oct 30 19:45:21.079390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.078799    6777 scope.go:122] "RemoveContainer" containerID="73c7e8b8d2bea15f4ad3939dc54c5d2052a842a6bfac127e8ee551ce8374d96f"
Oct 30 19:45:21.081015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.080981    6777 scope.go:122] "RemoveContainer" containerID="68776ad904170742bff1e3a9a001398c886995485d717122cea279264440225e"
Oct 30 19:45:21.081434 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:21.081404    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85_odh-ai-gateway-infra(37631bd2-5738-47fe-93d2-df713a302ff2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" podUID="37631bd2-5738-47fe-93d2-df713a302ff2"
Oct 30 19:45:21.084606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.084560    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" event={"ID":"6ec3359f-38ad-431c-9d07-cab499ec4598","Type":"ContainerStarted","Data":"7d84632128cee5638bd771dd00c31ea79dbaa2116166c449d2c3d12cf1c1278d"}
Oct 30 19:45:21.089050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.089007    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" event={"ID":"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37","Type":"ContainerStarted","Data":"dcfd6c86504e77f124101741d637c3ded774385bc20a428efba71dc794a6edee"}
Oct 30 19:45:21.094459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.094433    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"]
Oct 30 19:45:21.104876 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.104845    6777 scope.go:122] "RemoveContainer" containerID="b4f6dd2afc5db1bc95bb0dc506c67afff19d71190c7e6178f170c89c5a2a0fce"
Oct 30 19:45:21.105802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.105774    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"]
Oct 30 19:45:21.110149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.110115    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8e6762-29889825-jb7xc"]
Oct 30 19:45:21.142904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.142870    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"]
Oct 30 19:45:21.331961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.331857    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:21.399017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.398690    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debc150f-bdfc-401e-876f-f110044f94b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debc150f-bdfc-401e-876f-f110044f94b4-maas-api-service-ca\") pod \"debc150f-bdfc-401e-876f-f110044f94b4\" (UID: \"debc150f-bdfc-401e-876f-f110044f94b4\") "
Oct 30 19:45:21.399017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.398764    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debc150f-bdfc-401e-876f-f110044f94b4-kube-api-access-7l58k\" (UniqueName: \"kubernetes.io/projected/debc150f-bdfc-401e-876f-f110044f94b4-kube-api-access-7l58k\") pod \"debc150f-bdfc-401e-876f-f110044f94b4\" (UID: \"debc150f-bdfc-401e-876f-f110044f94b4\") "
Oct 30 19:45:21.399626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.399569    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debc150f-bdfc-401e-876f-f110044f94b4-maas-api-service-ca" pod "debc150f-bdfc-401e-876f-f110044f94b4" (UID: "debc150f-bdfc-401e-876f-f110044f94b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.400538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.399837    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debc150f-bdfc-401e-876f-f110044f94b4-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.422577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.412735    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debc150f-bdfc-401e-876f-f110044f94b4-kube-api-access-7l58k" pod "debc150f-bdfc-401e-876f-f110044f94b4" (UID: "debc150f-bdfc-401e-876f-f110044f94b4"). InnerVolumeSpecName "kube-api-access-7l58k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.502223 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.502182    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l58k\" (UniqueName: \"kubernetes.io/projected/debc150f-bdfc-401e-876f-f110044f94b4-kube-api-access-7l58k\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.610672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.610596    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:21.735135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.728991    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab87e647-dc82-4dbf-b754-45e414878aeb-kube-api-access-rvj55\" (UniqueName: \"kubernetes.io/projected/ab87e647-dc82-4dbf-b754-45e414878aeb-kube-api-access-rvj55\") pod \"ab87e647-dc82-4dbf-b754-45e414878aeb\" (UID: \"ab87e647-dc82-4dbf-b754-45e414878aeb\") "
Oct 30 19:45:21.735135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.729293    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab87e647-dc82-4dbf-b754-45e414878aeb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab87e647-dc82-4dbf-b754-45e414878aeb-maas-api-service-ca\") pod \"ab87e647-dc82-4dbf-b754-45e414878aeb\" (UID: \"ab87e647-dc82-4dbf-b754-45e414878aeb\") "
Oct 30 19:45:21.735135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.729819    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab87e647-dc82-4dbf-b754-45e414878aeb-maas-api-service-ca" pod "ab87e647-dc82-4dbf-b754-45e414878aeb" (UID: "ab87e647-dc82-4dbf-b754-45e414878aeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.735135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.730269    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab87e647-dc82-4dbf-b754-45e414878aeb-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.740833 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.738263    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab87e647-dc82-4dbf-b754-45e414878aeb-kube-api-access-rvj55" pod "ab87e647-dc82-4dbf-b754-45e414878aeb" (UID: "ab87e647-dc82-4dbf-b754-45e414878aeb"). InnerVolumeSpecName "kube-api-access-rvj55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.831907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:21.831873    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvj55\" (UniqueName: \"kubernetes.io/projected/ab87e647-dc82-4dbf-b754-45e414878aeb-kube-api-access-rvj55\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.113747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.113709    6777 generic.go:356] "Generic (PLEG): container finished" podID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerID="7d84632128cee5638bd771dd00c31ea79dbaa2116166c449d2c3d12cf1c1278d" exitCode=60
Oct 30 19:45:22.116484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.116454    6777 generic.go:356] "Generic (PLEG): container finished" podID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerID="dcfd6c86504e77f124101741d637c3ded774385bc20a428efba71dc794a6edee" exitCode=60
Oct 30 19:45:22.118641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.118363    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"
Oct 30 19:45:22.118641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.118630    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694b1588-e272-4115-ae41-908654b72649" path="/var/lib/kubelet/pods/694b1588-e272-4115-ae41-908654b72649/volumes"
Oct 30 19:45:22.119275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.119257    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" path="/var/lib/kubelet/pods/6fe76d57-03af-4460-b0ca-84ab80ef060b/volumes"
Oct 30 19:45:22.120101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.120034    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" event={"ID":"6ec3359f-38ad-431c-9d07-cab499ec4598","Type":"ContainerDied","Data":"7d84632128cee5638bd771dd00c31ea79dbaa2116166c449d2c3d12cf1c1278d"}
Oct 30 19:45:22.120101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.120093    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" event={"ID":"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37","Type":"ContainerDied","Data":"dcfd6c86504e77f124101741d637c3ded774385bc20a428efba71dc794a6edee"}
Oct 30 19:45:22.120299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.120115    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2" event={"ID":"ab87e647-dc82-4dbf-b754-45e414878aeb","Type":"ContainerDied","Data":"a3c45278de65eba53c7b82be734c7eb9d609cfc8586d1f909995bd429f8e771d"}
Oct 30 19:45:22.120299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.120141    6777 scope.go:122] "RemoveContainer" containerID="bd72ac9ed3eaa55f4560413e46aab15d07df335dcd889b6e3a525a5b95d8c599"
Oct 30 19:45:22.120830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.120624    6777 scope.go:122] "RemoveContainer" containerID="7d84632128cee5638bd771dd00c31ea79dbaa2116166c449d2c3d12cf1c1278d"
Oct 30 19:45:22.120939 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:22.120916    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm_odh-ai-gateway-infra(6ec3359f-38ad-431c-9d07-cab499ec4598)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598"
Oct 30 19:45:22.121173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.121154    6777 scope.go:122] "RemoveContainer" containerID="dcfd6c86504e77f124101741d637c3ded774385bc20a428efba71dc794a6edee"
Oct 30 19:45:22.121516 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:22.121488    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4_odh-ai-gateway-infra(b1a5a0ed-186c-493d-a8c5-ac70ca38fa37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37"
Oct 30 19:45:22.124380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.124357    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z" event={"ID":"debc150f-bdfc-401e-876f-f110044f94b4","Type":"ContainerDied","Data":"ad7b68f6d3987b6f38a6c18c5fe766686a08c608e1350e57632bcd79b2e284ea"}
Oct 30 19:45:22.124507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.124451    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"
Oct 30 19:45:22.133430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.133318    6777 generic.go:356] "Generic (PLEG): container finished" podID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerID="7f93bf8a074445ca6a6cc85b3b6e914c3a6054ccc9875a9fcebeb11a80096168" exitCode=60
Oct 30 19:45:22.133430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.133415    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" event={"ID":"e3d8c1a8-a99c-401a-9f79-9657922c2319","Type":"ContainerDied","Data":"7f93bf8a074445ca6a6cc85b3b6e914c3a6054ccc9875a9fcebeb11a80096168"}
Oct 30 19:45:22.135715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.135689    6777 scope.go:122] "RemoveContainer" containerID="7f93bf8a074445ca6a6cc85b3b6e914c3a6054ccc9875a9fcebeb11a80096168"
Oct 30 19:45:22.138612 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:22.136213    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff_odh-ai-gateway-infra(e3d8c1a8-a99c-401a-9f79-9657922c2319)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319"
Oct 30 19:45:22.152548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.151491    6777 scope.go:122] "RemoveContainer" containerID="7e4250d4809158356eebe2ac99898214e75f51b2826e09668dc9f8b8a65d56ab"
Oct 30 19:45:22.155422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.155318    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"]
Oct 30 19:45:22.176564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.176449    6777 scope.go:122] "RemoveContainer" containerID="28ae62994c1997a106a37cc1dd04f3672ae87176debb29524010455a974c43b5"
Oct 30 19:45:22.202455 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.202017    6777 scope.go:122] "RemoveContainer" containerID="d181af62f7ef15af9cef19e914b308b08ae8e153d42fd0a0ff6aaa79f596296f"
Oct 30 19:45:22.226117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.220105    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"]
Oct 30 19:45:22.266141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.264438    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"]
Oct 30 19:45:22.266331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.266311    6777 scope.go:122] "RemoveContainer" containerID="f65424ec0f42803044e15660b6ea9457d4ff4fda1ba98cddc6062758f624e8db"
Oct 30 19:45:22.277249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.277215    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"]
Oct 30 19:45:22.293532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.293488    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aa217-29889825-vgw2z"]
Oct 30 19:45:22.318805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.318775    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"]
Oct 30 19:45:22.320244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.320211    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ddb6ea-29889825-kzww2"]
Oct 30 19:45:22.377079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.376265    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:22.485022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.484925    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37631bd2-5738-47fe-93d2-df713a302ff2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37631bd2-5738-47fe-93d2-df713a302ff2-maas-api-service-ca\") pod \"37631bd2-5738-47fe-93d2-df713a302ff2\" (UID: \"37631bd2-5738-47fe-93d2-df713a302ff2\") "
Oct 30 19:45:22.485856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.485823    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37631bd2-5738-47fe-93d2-df713a302ff2-kube-api-access-ftmdc\" (UniqueName: \"kubernetes.io/projected/37631bd2-5738-47fe-93d2-df713a302ff2-kube-api-access-ftmdc\") pod \"37631bd2-5738-47fe-93d2-df713a302ff2\" (UID: \"37631bd2-5738-47fe-93d2-df713a302ff2\") "
Oct 30 19:45:22.489121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.489033    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37631bd2-5738-47fe-93d2-df713a302ff2-maas-api-service-ca" pod "37631bd2-5738-47fe-93d2-df713a302ff2" (UID: "37631bd2-5738-47fe-93d2-df713a302ff2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:22.505117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.504921    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37631bd2-5738-47fe-93d2-df713a302ff2-kube-api-access-ftmdc" pod "37631bd2-5738-47fe-93d2-df713a302ff2" (UID: "37631bd2-5738-47fe-93d2-df713a302ff2"). InnerVolumeSpecName "kube-api-access-ftmdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:22.560552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.560517    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:22.588285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.588240    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-maas-api-service-ca\") pod \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\" (UID: \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\") "
Oct 30 19:45:22.588499 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.588380    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-kube-api-access-kcb5h\" (UniqueName: \"kubernetes.io/projected/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-kube-api-access-kcb5h\") pod \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\" (UID: \"c9c7be5e-5f69-4a56-b45e-64ded354d9c2\") "
Oct 30 19:45:22.588662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.588628    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-maas-api-service-ca" pod "c9c7be5e-5f69-4a56-b45e-64ded354d9c2" (UID: "c9c7be5e-5f69-4a56-b45e-64ded354d9c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:22.589475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.589441    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.589568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.589474    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftmdc\" (UniqueName: \"kubernetes.io/projected/37631bd2-5738-47fe-93d2-df713a302ff2-kube-api-access-ftmdc\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.589568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.589493    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37631bd2-5738-47fe-93d2-df713a302ff2-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.591189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.591161    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-kube-api-access-kcb5h" pod "c9c7be5e-5f69-4a56-b45e-64ded354d9c2" (UID: "c9c7be5e-5f69-4a56-b45e-64ded354d9c2"). InnerVolumeSpecName "kube-api-access-kcb5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:22.691541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:22.691491    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcb5h\" (UniqueName: \"kubernetes.io/projected/c9c7be5e-5f69-4a56-b45e-64ded354d9c2-kube-api-access-kcb5h\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:23.149651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.149628    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"
Oct 30 19:45:23.149795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.149666    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9" event={"ID":"c9c7be5e-5f69-4a56-b45e-64ded354d9c2","Type":"ContainerDied","Data":"32c41691c1a377719ed43c8215439299ea2956edf1914f33f7de9a836f45550e"}
Oct 30 19:45:23.149795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.149724    6777 scope.go:122] "RemoveContainer" containerID="2c3ff24fc0a03b28c123281d405dc07f844d563422f932ff3ea7b627d0dc40b0"
Oct 30 19:45:23.151536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.151503    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"
Oct 30 19:45:23.151648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.151533    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85" event={"ID":"37631bd2-5738-47fe-93d2-df713a302ff2","Type":"ContainerDied","Data":"6644f02d29a06209788515f18e2f42843895ef8e040767b555f9881fd8cdbed7"}
Oct 30 19:45:23.164611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.164587    6777 scope.go:122] "RemoveContainer" containerID="73c7e8b8d2bea15f4ad3939dc54c5d2052a842a6bfac127e8ee551ce8374d96f"
Oct 30 19:45:23.352484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.349246    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"]
Oct 30 19:45:23.352484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.352273    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8f3ff4-29889825-pgj85"]
Oct 30 19:45:23.357662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.357628    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"]
Oct 30 19:45:23.362272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.362238    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f5613a-29889825-kxjg9"]
Oct 30 19:45:23.476668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.476482    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:23.549611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.549574    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d8c1a8-a99c-401a-9f79-9657922c2319-kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/e3d8c1a8-a99c-401a-9f79-9657922c2319-kube-api-access-w8h4h\") pod \"e3d8c1a8-a99c-401a-9f79-9657922c2319\" (UID: \"e3d8c1a8-a99c-401a-9f79-9657922c2319\") "
Oct 30 19:45:23.550257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.550231    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d8c1a8-a99c-401a-9f79-9657922c2319-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8c1a8-a99c-401a-9f79-9657922c2319-maas-api-service-ca\") pod \"e3d8c1a8-a99c-401a-9f79-9657922c2319\" (UID: \"e3d8c1a8-a99c-401a-9f79-9657922c2319\") "
Oct 30 19:45:23.554946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.554918    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d8c1a8-a99c-401a-9f79-9657922c2319-maas-api-service-ca" pod "e3d8c1a8-a99c-401a-9f79-9657922c2319" (UID: "e3d8c1a8-a99c-401a-9f79-9657922c2319"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:23.556083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.556040    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d8c1a8-a99c-401a-9f79-9657922c2319-kube-api-access-w8h4h" pod "e3d8c1a8-a99c-401a-9f79-9657922c2319" (UID: "e3d8c1a8-a99c-401a-9f79-9657922c2319"). InnerVolumeSpecName "kube-api-access-w8h4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:23.603610 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.603581    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:23.609405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.609376    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:23.652411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.652372    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec3359f-38ad-431c-9d07-cab499ec4598-kube-api-access-q4fth\" (UniqueName: \"kubernetes.io/projected/6ec3359f-38ad-431c-9d07-cab499ec4598-kube-api-access-q4fth\") pod \"6ec3359f-38ad-431c-9d07-cab499ec4598\" (UID: \"6ec3359f-38ad-431c-9d07-cab499ec4598\") "
Oct 30 19:45:23.653104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.653078    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-kube-api-access-6mh42\" (UniqueName: \"kubernetes.io/projected/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-kube-api-access-6mh42\") pod \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\" (UID: \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\") "
Oct 30 19:45:23.653716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.653695    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-maas-api-service-ca\") pod \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\" (UID: \"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37\") "
Oct 30 19:45:23.654524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.654503    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec3359f-38ad-431c-9d07-cab499ec4598-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3359f-38ad-431c-9d07-cab499ec4598-maas-api-service-ca\") pod \"6ec3359f-38ad-431c-9d07-cab499ec4598\" (UID: \"6ec3359f-38ad-431c-9d07-cab499ec4598\") "
Oct 30 19:45:23.656035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.656004    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/e3d8c1a8-a99c-401a-9f79-9657922c2319-kube-api-access-w8h4h\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:23.656224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.656039    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8c1a8-a99c-401a-9f79-9657922c2319-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:23.656224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.654199    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-maas-api-service-ca" pod "b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" (UID: "b1a5a0ed-186c-493d-a8c5-ac70ca38fa37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:23.656224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.655018    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec3359f-38ad-431c-9d07-cab499ec4598-maas-api-service-ca" pod "6ec3359f-38ad-431c-9d07-cab499ec4598" (UID: "6ec3359f-38ad-431c-9d07-cab499ec4598"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:23.658942 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.658553    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec3359f-38ad-431c-9d07-cab499ec4598-kube-api-access-q4fth" pod "6ec3359f-38ad-431c-9d07-cab499ec4598" (UID: "6ec3359f-38ad-431c-9d07-cab499ec4598"). InnerVolumeSpecName "kube-api-access-q4fth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:23.673899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.673800    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-kube-api-access-6mh42" pod "b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" (UID: "b1a5a0ed-186c-493d-a8c5-ac70ca38fa37"). InnerVolumeSpecName "kube-api-access-6mh42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:23.757624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.757587    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3359f-38ad-431c-9d07-cab499ec4598-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:23.757624 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.757622    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4fth\" (UniqueName: \"kubernetes.io/projected/6ec3359f-38ad-431c-9d07-cab499ec4598-kube-api-access-q4fth\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:23.757938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.757638    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mh42\" (UniqueName: \"kubernetes.io/projected/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-kube-api-access-6mh42\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:23.757938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:23.757650    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:24.263083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.262880    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" path="/var/lib/kubelet/pods/37631bd2-5738-47fe-93d2-df713a302ff2/volumes"
Oct 30 19:45:24.289705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.274219    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" path="/var/lib/kubelet/pods/ab87e647-dc82-4dbf-b754-45e414878aeb/volumes"
Oct 30 19:45:24.289705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.275021    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" path="/var/lib/kubelet/pods/c9c7be5e-5f69-4a56-b45e-64ded354d9c2/volumes"
Oct 30 19:45:24.325921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.325881    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"
Oct 30 19:45:24.328981 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.328944    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debc150f-bdfc-401e-876f-f110044f94b4" path="/var/lib/kubelet/pods/debc150f-bdfc-401e-876f-f110044f94b4/volumes"
Oct 30 19:45:24.335696 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.335660    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"
Oct 30 19:45:24.337009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.336976    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff" event={"ID":"e3d8c1a8-a99c-401a-9f79-9657922c2319","Type":"ContainerDied","Data":"a0bcc1bc7865d7a8c4eb0abbcdb6a353274ac3456abfacedde8a9ac4cca280b3"}
Oct 30 19:45:24.337324 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.337300    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm" event={"ID":"6ec3359f-38ad-431c-9d07-cab499ec4598","Type":"ContainerDied","Data":"89b7be6d0c76003a64c9e168b49ced7dacb6c2e4e5fec71b375059e12b1ad658"}
Oct 30 19:45:24.337567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.337549    6777 scope.go:122] "RemoveContainer" containerID="7f93bf8a074445ca6a6cc85b3b6e914c3a6054ccc9875a9fcebeb11a80096168"
Oct 30 19:45:24.339980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.337839    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4" event={"ID":"b1a5a0ed-186c-493d-a8c5-ac70ca38fa37","Type":"ContainerDied","Data":"57cf8c93d0cc394e3b7d9abd1f5bf3d1d17002fdb5c033f2e1a5f368e4a866d8"}
Oct 30 19:45:24.339980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.337902    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"
Oct 30 19:45:24.797738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.797690    6777 scope.go:122] "RemoveContainer" containerID="7d84632128cee5638bd771dd00c31ea79dbaa2116166c449d2c3d12cf1c1278d"
Oct 30 19:45:24.801394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.801370    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"]
Oct 30 19:45:24.810162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.810120    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-618d93-29889825-ttxff"]
Oct 30 19:45:24.839352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.831632    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"]
Oct 30 19:45:24.839352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.836148    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e3586f-29889825-bccc4"]
Oct 30 19:45:24.839352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.837866    6777 scope.go:122] "RemoveContainer" containerID="dcfd6c86504e77f124101741d637c3ded774385bc20a428efba71dc794a6edee"
Oct 30 19:45:24.847795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.847568    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"]
Oct 30 19:45:24.855389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.855321    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e50648-29889825-zcdbm"]
Oct 30 19:45:24.887168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:24.887087    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:25.739081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:25.737796    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"]
Oct 30 19:45:25.748043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:25.748007    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"]
Oct 30 19:45:26.236078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:26.235694    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" path="/var/lib/kubelet/pods/6ec3359f-38ad-431c-9d07-cab499ec4598/volumes"
Oct 30 19:45:26.236615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:26.236546    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" path="/var/lib/kubelet/pods/b1a5a0ed-186c-493d-a8c5-ac70ca38fa37/volumes"
Oct 30 19:45:26.253680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:26.237788    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" path="/var/lib/kubelet/pods/e3d8c1a8-a99c-401a-9f79-9657922c2319/volumes"
Oct 30 19:45:28.970152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:28.970116    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"]
Oct 30 19:45:28.997110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:28.989670    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"]
Oct 30 19:45:29.606464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:29.606430    6777 generic.go:356] "Generic (PLEG): container finished" podID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerID="0e3232805a6b390d0e668734d511446ffb2394c76d5dc6bb676bc99d3fdbc4d0" exitCode=0
Oct 30 19:45:29.606682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:29.606465    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" event={"ID":"8e937cd6-3dcd-4309-9494-3df69a56d551","Type":"ContainerDied","Data":"0e3232805a6b390d0e668734d511446ffb2394c76d5dc6bb676bc99d3fdbc4d0"}
Oct 30 19:45:29.607171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:29.607149    6777 scope.go:122] "RemoveContainer" containerID="0e3232805a6b390d0e668734d511446ffb2394c76d5dc6bb676bc99d3fdbc4d0"
Oct 30 19:45:29.609199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:29.609170    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:29.609318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:29.609256    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:29.609318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:29.609303    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:30.629914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:30.629875    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" event={"ID":"8e937cd6-3dcd-4309-9494-3df69a56d551","Type":"ContainerStarted","Data":"3ec8ee47cd8524db374f6f07f4fbc55c45d389c1fdb163a426893b4c38690811"}
Oct 30 19:45:30.738841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:30.738791    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7" podUID="70fd4577-7428-4f87-96f6-5e84954bbaa2" containerName="payload-processing" containerID="cri-o://749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc" gracePeriod=25
Oct 30 19:45:30.755227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:30.752156    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc" podUID="78d9f55b-bf47-40ca-8715-0fd1cbbbb893" containerName="payload-pre-processing" containerID="cri-o://895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495" gracePeriod=25
Oct 30 19:45:31.125944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.125910    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:45:31.181648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.181417    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:45:31.182705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.182378    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:45:31.182705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.182434    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Oct 30 19:45:31.215359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.215323    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:45:31.301368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.301336    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70fd4577-7428-4f87-96f6-5e84954bbaa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fd4577-7428-4f87-96f6-5e84954bbaa2-plugins-config-volume\") pod \"70fd4577-7428-4f87-96f6-5e84954bbaa2\" (UID: \"70fd4577-7428-4f87-96f6-5e84954bbaa2\") "
Oct 30 19:45:31.301589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.301389    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fd4577-7428-4f87-96f6-5e84954bbaa2-kube-api-access-tz4qf\" (UniqueName: \"kubernetes.io/projected/70fd4577-7428-4f87-96f6-5e84954bbaa2-kube-api-access-tz4qf\") pod \"70fd4577-7428-4f87-96f6-5e84954bbaa2\" (UID: \"70fd4577-7428-4f87-96f6-5e84954bbaa2\") "
Oct 30 19:45:31.301800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.301777    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70fd4577-7428-4f87-96f6-5e84954bbaa2-plugins-config-volume" pod "70fd4577-7428-4f87-96f6-5e84954bbaa2" (UID: "70fd4577-7428-4f87-96f6-5e84954bbaa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:31.302325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.302303    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70fd4577-7428-4f87-96f6-5e84954bbaa2-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:31.303883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.303842    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fd4577-7428-4f87-96f6-5e84954bbaa2-kube-api-access-tz4qf" pod "70fd4577-7428-4f87-96f6-5e84954bbaa2" (UID: "70fd4577-7428-4f87-96f6-5e84954bbaa2"). InnerVolumeSpecName "kube-api-access-tz4qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:31.403816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.403781    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-kube-api-access-fjhlc\" (UniqueName: \"kubernetes.io/projected/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-kube-api-access-fjhlc\") pod \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\" (UID: \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\") "
Oct 30 19:45:31.404133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.404118    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-plugins-config-volume\") pod \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\" (UID: \"78d9f55b-bf47-40ca-8715-0fd1cbbbb893\") "
Oct 30 19:45:31.404574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.404548    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-plugins-config-volume" pod "78d9f55b-bf47-40ca-8715-0fd1cbbbb893" (UID: "78d9f55b-bf47-40ca-8715-0fd1cbbbb893"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:31.404762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.404746    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4qf\" (UniqueName: \"kubernetes.io/projected/70fd4577-7428-4f87-96f6-5e84954bbaa2-kube-api-access-tz4qf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:31.404829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.404767    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:31.406213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.406190    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-kube-api-access-fjhlc" pod "78d9f55b-bf47-40ca-8715-0fd1cbbbb893" (UID: "78d9f55b-bf47-40ca-8715-0fd1cbbbb893"). InnerVolumeSpecName "kube-api-access-fjhlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:31.515463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.515363    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjhlc\" (UniqueName: \"kubernetes.io/projected/78d9f55b-bf47-40ca-8715-0fd1cbbbb893-kube-api-access-fjhlc\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:31.640402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.640364    6777 generic.go:356] "Generic (PLEG): container finished" podID="70fd4577-7428-4f87-96f6-5e84954bbaa2" containerID="749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc" exitCode=0
Oct 30 19:45:31.640899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.640424    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"
Oct 30 19:45:31.640899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.640422    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7" event={"ID":"70fd4577-7428-4f87-96f6-5e84954bbaa2","Type":"ContainerDied","Data":"749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc"}
Oct 30 19:45:31.640899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.640553    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7" event={"ID":"70fd4577-7428-4f87-96f6-5e84954bbaa2","Type":"ContainerDied","Data":"1d89ef4554b05da58919a13c38eda27f8e9934dcc55d10ba1162e8fca60d7a80"}
Oct 30 19:45:31.640899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.640576    6777 scope.go:122] "RemoveContainer" containerID="749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc"
Oct 30 19:45:31.642149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.642127    6777 generic.go:356] "Generic (PLEG): container finished" podID="78d9f55b-bf47-40ca-8715-0fd1cbbbb893" containerID="895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495" exitCode=0
Oct 30 19:45:31.642276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.642164    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"
Oct 30 19:45:31.642276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.642257    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc" event={"ID":"78d9f55b-bf47-40ca-8715-0fd1cbbbb893","Type":"ContainerDied","Data":"895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495"}
Oct 30 19:45:31.642395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.642299    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc" event={"ID":"78d9f55b-bf47-40ca-8715-0fd1cbbbb893","Type":"ContainerDied","Data":"2f9044986715e23c6ac5c94f5c306447baeb4519b23dabf73d36dcec537d9829"}
Oct 30 19:45:31.679678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.679652    6777 scope.go:122] "RemoveContainer" containerID="749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc"
Oct 30 19:45:31.680075 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:31.680026    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc\": container with ID starting with 749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc not found: ID does not exist" containerID="749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc"
Oct 30 19:45:31.680170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.680086    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc"} err="failed to get container status \"749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc\": rpc error: code = NotFound desc = could not find container \"749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc\": container with ID starting with 749372384e7d38896d415f2a4a854a2988fce478fb0882dfa85129a20a3979fc not found: ID does not exist"
Oct 30 19:45:31.680170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.680111    6777 scope.go:122] "RemoveContainer" containerID="895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495"
Oct 30 19:45:31.691457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.691431    6777 scope.go:122] "RemoveContainer" containerID="895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495"
Oct 30 19:45:31.691786 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:31.691764    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495\": container with ID starting with 895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495 not found: ID does not exist" containerID="895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495"
Oct 30 19:45:31.691873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.691793    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495"} err="failed to get container status \"895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495\": rpc error: code = NotFound desc = could not find container \"895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495\": container with ID starting with 895488c56c922aee525f82c98b174bc9e2b691f7bf3e761ab4ee3577bbef5495 not found: ID does not exist"
Oct 30 19:45:31.701909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.700547    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"]
Oct 30 19:45:31.705690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.705659    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f5613a-75fbd77c88-vvvk7"]
Oct 30 19:45:31.720793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.720758    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"]
Oct 30 19:45:31.725411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:31.725381    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5613a-8d74b8bd7-k5stc"]
Oct 30 19:45:32.192137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:32.191344    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:45:32.211098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:32.205245    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Oct 30 19:45:32.211098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:32.206142    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70fd4577-7428-4f87-96f6-5e84954bbaa2" path="/var/lib/kubelet/pods/70fd4577-7428-4f87-96f6-5e84954bbaa2/volumes"
Oct 30 19:45:32.211098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:32.207474    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d9f55b-bf47-40ca-8715-0fd1cbbbb893" path="/var/lib/kubelet/pods/78d9f55b-bf47-40ca-8715-0fd1cbbbb893/volumes"
Oct 30 19:45:33.197087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:33.196374    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:45:33.197087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:33.196442    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Oct 30 19:45:33.232084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:33.227167    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:45:33.973139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:33.971144    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2" podUID="13176ae0-fa1c-4020-b9aa-cbdbc28ac07f" containerName="payload-processing" containerID="cri-o://8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e" gracePeriod=25
Oct 30 19:45:33.991130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:33.991027    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2" podUID="d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd" containerName="payload-pre-processing" containerID="cri-o://7483689ed74efc540190fa26fb34964892eda57755866c5a1ede53d4ae3b9059" gracePeriod=25
Oct 30 19:45:34.180083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:34.179418    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:45:34.180083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:34.179477    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Oct 30 19:45:34.692641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:34.692552    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:45:34.956402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:34.908021    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-plugins-config-volume\") pod \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\" (UID: \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\") "
Oct 30 19:45:34.956402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:34.908238    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-kube-api-access-plscx\" (UniqueName: \"kubernetes.io/projected/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-kube-api-access-plscx\") pod \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\" (UID: \"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f\") "
Oct 30 19:45:34.956402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:34.928352    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-plugins-config-volume" pod "13176ae0-fa1c-4020-b9aa-cbdbc28ac07f" (UID: "13176ae0-fa1c-4020-b9aa-cbdbc28ac07f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:35.019128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.017793    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-kube-api-access-plscx" pod "13176ae0-fa1c-4020-b9aa-cbdbc28ac07f" (UID: "13176ae0-fa1c-4020-b9aa-cbdbc28ac07f"). InnerVolumeSpecName "kube-api-access-plscx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:35.062079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.021026    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plscx\" (UniqueName: \"kubernetes.io/projected/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-kube-api-access-plscx\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:35.062079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.047468    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:35.082088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.078206    6777 generic.go:356] "Generic (PLEG): container finished" podID="13176ae0-fa1c-4020-b9aa-cbdbc28ac07f" containerID="8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e" exitCode=0
Oct 30 19:45:35.082088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.078524    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"
Oct 30 19:45:35.082088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.080264    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2" event={"ID":"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f","Type":"ContainerDied","Data":"8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e"}
Oct 30 19:45:35.082088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.080348    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2" event={"ID":"13176ae0-fa1c-4020-b9aa-cbdbc28ac07f","Type":"ContainerDied","Data":"a863869c49777dc65e517b0271a3a8f08a9e24d6d8e31f715da01f2183cd4ad1"}
Oct 30 19:45:35.082088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.080382    6777 scope.go:122] "RemoveContainer" containerID="8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e"
Oct 30 19:45:35.180575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.180536    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:45:35.190210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.190158    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Oct 30 19:45:35.214091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.196847    6777 generic.go:356] "Generic (PLEG): container finished" podID="d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd" containerID="7483689ed74efc540190fa26fb34964892eda57755866c5a1ede53d4ae3b9059" exitCode=0
Oct 30 19:45:35.214091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.196876    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2" event={"ID":"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd","Type":"ContainerDied","Data":"7483689ed74efc540190fa26fb34964892eda57755866c5a1ede53d4ae3b9059"}
Oct 30 19:45:35.427651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.427622    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:45:35.429210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.429187    6777 scope.go:122] "RemoveContainer" containerID="8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e"
Oct 30 19:45:35.429553 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:35.429534    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e\": container with ID starting with 8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e not found: ID does not exist" containerID="8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e"
Oct 30 19:45:35.429616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.429562    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e"} err="failed to get container status \"8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e\": rpc error: code = NotFound desc = could not find container \"8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e\": container with ID starting with 8e9e0f598c2864f2c20ed16edc0f87666e4ed533d75e9f18f6e19490f8ce8c6e not found: ID does not exist"
Oct 30 19:45:35.525226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.525144    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"]
Oct 30 19:45:35.528622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.528593    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-5cc45dc6bd-t67z2"]
Oct 30 19:45:35.533025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.532999    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-plugins-config-volume\") pod \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\" (UID: \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\") "
Oct 30 19:45:35.533236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.533218    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-kube-api-access-tt8rk\" (UniqueName: \"kubernetes.io/projected/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-kube-api-access-tt8rk\") pod \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\" (UID: \"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd\") "
Oct 30 19:45:35.533430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.533408    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-plugins-config-volume" pod "d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd" (UID: "d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:35.534105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.534049    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:35.535570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.535543    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-kube-api-access-tt8rk" pod "d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd" (UID: "d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd"). InnerVolumeSpecName "kube-api-access-tt8rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:35.636229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:35.636184    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8rk\" (UniqueName: \"kubernetes.io/projected/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd-kube-api-access-tt8rk\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:36.119840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:36.119803    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13176ae0-fa1c-4020-b9aa-cbdbc28ac07f" path="/var/lib/kubelet/pods/13176ae0-fa1c-4020-b9aa-cbdbc28ac07f/volumes"
Oct 30 19:45:36.195084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:36.194806    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:45:36.195084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:36.194912    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Oct 30 19:45:36.334078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:36.330562    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2" event={"ID":"d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd","Type":"ContainerDied","Data":"fe44134c149d1773a91a9fa646220cbd0a25543102050ee129112ec0c9aa1e21"}
Oct 30 19:45:36.334078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:36.330636    6777 scope.go:122] "RemoveContainer" containerID="7483689ed74efc540190fa26fb34964892eda57755866c5a1ede53d4ae3b9059"
Oct 30 19:45:36.334078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:36.330756    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"
Oct 30 19:45:36.405140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:36.401778    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"]
Oct 30 19:45:36.463088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:36.459727    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-584b67dfcf-k97v2"]
Oct 30 19:45:37.179367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:37.179243    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:45:37.179367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:37.179304    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Oct 30 19:45:38.122023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:38.121985    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd" path="/var/lib/kubelet/pods/d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd/volumes"
Oct 30 19:45:38.183593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:38.183558    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:45:38.184791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:38.184767    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:45:40.043157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043118    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"]
Oct 30 19:45:40.043895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043880    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerName="cleanup"
Oct 30 19:45:40.043949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043896    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerName="cleanup"
Oct 30 19:45:40.043949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043908    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerName="cleanup"
Oct 30 19:45:40.043949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043913    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerName="cleanup"
Oct 30 19:45:40.043949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043940    6777 cpu_manager.go:414] "containerMap: removing container" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" containerName="cleanup"
Oct 30 19:45:40.043949 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043947    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043955    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043962    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043967    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd" containerName="payload-pre-processing"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043973    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd" containerName="payload-pre-processing"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043980    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043985    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.043997    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044001    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044011    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044026    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044042    6777 cpu_manager.go:414] "containerMap: removing container" podUID="70fd4577-7428-4f87-96f6-5e84954bbaa2" containerName="payload-processing"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044046    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70fd4577-7428-4f87-96f6-5e84954bbaa2" containerName="payload-processing"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044075    6777 cpu_manager.go:414] "containerMap: removing container" podUID="debc150f-bdfc-401e-876f-f110044f94b4" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044082    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debc150f-bdfc-401e-876f-f110044f94b4" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044092    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044098    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044107    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044114    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044125    6777 cpu_manager.go:414] "containerMap: removing container" podUID="694b1588-e272-4115-ae41-908654b72649" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044132    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694b1588-e272-4115-ae41-908654b72649" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044148    6777 cpu_manager.go:414] "containerMap: removing container" podUID="694b1588-e272-4115-ae41-908654b72649" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044155    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694b1588-e272-4115-ae41-908654b72649" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044164    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044173    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044194    6777 cpu_manager.go:414] "containerMap: removing container" podUID="debc150f-bdfc-401e-876f-f110044f94b4" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044201    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debc150f-bdfc-401e-876f-f110044f94b4" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044220    6777 cpu_manager.go:414] "containerMap: removing container" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044227    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044250    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044257    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerName="cleanup"
Oct 30 19:45:40.044251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044269    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044276    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044286    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044293    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044310    6777 cpu_manager.go:414] "containerMap: removing container" podUID="78d9f55b-bf47-40ca-8715-0fd1cbbbb893" containerName="payload-pre-processing"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044318    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d9f55b-bf47-40ca-8715-0fd1cbbbb893" containerName="payload-pre-processing"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044342    6777 cpu_manager.go:414] "containerMap: removing container" podUID="13176ae0-fa1c-4020-b9aa-cbdbc28ac07f" containerName="payload-processing"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044349    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13176ae0-fa1c-4020-b9aa-cbdbc28ac07f" containerName="payload-processing"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044360    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044367    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044392    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044401    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044413    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044419    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044674    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044687    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98bf8ff-e8c2-4600-ad27-06fd8e3d0bcd" containerName="payload-pre-processing"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044699    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044709    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044717    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044727    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="70fd4577-7428-4f87-96f6-5e84954bbaa2" containerName="payload-processing"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044735    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="debc150f-bdfc-401e-876f-f110044f94b4" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044743    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044751    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044760    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="694b1588-e272-4115-ae41-908654b72649" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044767    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044774    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="694b1588-e272-4115-ae41-908654b72649" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044781    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044788    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044798    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="debc150f-bdfc-401e-876f-f110044f94b4" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044806    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044814    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec3359f-38ad-431c-9d07-cab499ec4598" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044824    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044830    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044837    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d9f55b-bf47-40ca-8715-0fd1cbbbb893" containerName="payload-pre-processing"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044853    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="13176ae0-fa1c-4020-b9aa-cbdbc28ac07f" containerName="payload-processing"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044866    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044895    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a5a0ed-186c-493d-a8c5-ac70ca38fa37" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044906    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerName="cleanup"
Oct 30 19:45:40.045522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.044913    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" containerName="cleanup"
Oct 30 19:45:40.066697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.066668    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:40.084680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.080600    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"]
Oct 30 19:45:40.084680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.083936    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a844e\""
Oct 30 19:45:40.084680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.084045    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a844e-dockercfg-tppws\""
Oct 30 19:45:40.103993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.103955    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"]
Oct 30 19:45:40.104860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104841    6777 cpu_manager.go:414] "containerMap: removing container" podUID="694b1588-e272-4115-ae41-908654b72649" containerName="cleanup"
Oct 30 19:45:40.104930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104861    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694b1588-e272-4115-ae41-908654b72649" containerName="cleanup"
Oct 30 19:45:40.104930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104876    6777 cpu_manager.go:414] "containerMap: removing container" podUID="debc150f-bdfc-401e-876f-f110044f94b4" containerName="cleanup"
Oct 30 19:45:40.104930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104881    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debc150f-bdfc-401e-876f-f110044f94b4" containerName="cleanup"
Oct 30 19:45:40.104930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104893    6777 cpu_manager.go:414] "containerMap: removing container" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" containerName="cleanup"
Oct 30 19:45:40.104930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104898    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" containerName="cleanup"
Oct 30 19:45:40.105141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104937    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerName="cleanup"
Oct 30 19:45:40.105141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104943    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerName="cleanup"
Oct 30 19:45:40.105141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104955    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerName="cleanup"
Oct 30 19:45:40.105141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104961    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerName="cleanup"
Oct 30 19:45:40.105141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104990    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerName="cleanup"
Oct 30 19:45:40.105141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.104995    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d8c1a8-a99c-401a-9f79-9657922c2319" containerName="cleanup"
Oct 30 19:45:40.105141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.105141    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c7be5e-5f69-4a56-b45e-64ded354d9c2" containerName="cleanup"
Oct 30 19:45:40.105356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.105156    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="694b1588-e272-4115-ae41-908654b72649" containerName="cleanup"
Oct 30 19:45:40.105356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.105164    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe76d57-03af-4460-b0ca-84ab80ef060b" containerName="cleanup"
Oct 30 19:45:40.105356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.105170    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="debc150f-bdfc-401e-876f-f110044f94b4" containerName="cleanup"
Oct 30 19:45:40.105356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.105177    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="37631bd2-5738-47fe-93d2-df713a302ff2" containerName="cleanup"
Oct 30 19:45:40.105356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.105188    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab87e647-dc82-4dbf-b754-45e414878aeb" containerName="cleanup"
Oct 30 19:45:40.108159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.108136    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:40.127804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.127769    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"]
Oct 30 19:45:40.364318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.364222    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3590172e-2fb2-4488-a7b8-bd257dd73d91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z\" (UID: \"3590172e-2fb2-4488-a7b8-bd257dd73d91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:40.366138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.366103    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5\" (UID: \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:40.376216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.376180    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfqj\" (UniqueName: \"kubernetes.io/projected/3590172e-2fb2-4488-a7b8-bd257dd73d91-kube-api-access-bcfqj\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z\" (UID: \"3590172e-2fb2-4488-a7b8-bd257dd73d91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:40.411376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.402214    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9f2\" (UniqueName: \"kubernetes.io/projected/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-kube-api-access-hh9f2\") pod \"payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5\" (UID: \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:40.532197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.532138    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3590172e-2fb2-4488-a7b8-bd257dd73d91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z\" (UID: \"3590172e-2fb2-4488-a7b8-bd257dd73d91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:40.532470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.532449    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5\" (UID: \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:40.532736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.532717    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfqj\" (UniqueName: \"kubernetes.io/projected/3590172e-2fb2-4488-a7b8-bd257dd73d91-kube-api-access-bcfqj\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z\" (UID: \"3590172e-2fb2-4488-a7b8-bd257dd73d91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:40.532859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.532846    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9f2\" (UniqueName: \"kubernetes.io/projected/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-kube-api-access-hh9f2\") pod \"payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5\" (UID: \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:40.534397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.534369    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3590172e-2fb2-4488-a7b8-bd257dd73d91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z\" (UID: \"3590172e-2fb2-4488-a7b8-bd257dd73d91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:40.535406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.535384    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5\" (UID: \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:40.591086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.583050    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9f2\" (UniqueName: \"kubernetes.io/projected/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-kube-api-access-hh9f2\") pod \"payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5\" (UID: \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:40.591086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.587833    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfqj\" (UniqueName: \"kubernetes.io/projected/3590172e-2fb2-4488-a7b8-bd257dd73d91-kube-api-access-bcfqj\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z\" (UID: \"3590172e-2fb2-4488-a7b8-bd257dd73d91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:40.708155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.707437    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:40.737651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:40.737614    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:41.073709 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:41.072972    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3590172e_2fb2_4488_a7b8_bd257dd73d91.slice/crio-dfbe684fbeda85db32aef3e3739aadaf2540b08abb8b2e77bd2eba57ee2a8843 WatchSource:0}: Error finding container dfbe684fbeda85db32aef3e3739aadaf2540b08abb8b2e77bd2eba57ee2a8843: Status 404 returned error can't find the container with id dfbe684fbeda85db32aef3e3739aadaf2540b08abb8b2e77bd2eba57ee2a8843
Oct 30 19:45:41.076790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.076232    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"]
Oct 30 19:45:41.083011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.082978    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"]
Oct 30 19:45:41.526798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.526703    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z" event={"ID":"3590172e-2fb2-4488-a7b8-bd257dd73d91","Type":"ContainerStarted","Data":"e51819e5597deb4074c790dc499923e443b7f3bd5e9a0b55deccf4d1d4fb4177"}
Oct 30 19:45:41.526798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.526759    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z" event={"ID":"3590172e-2fb2-4488-a7b8-bd257dd73d91","Type":"ContainerStarted","Data":"dfbe684fbeda85db32aef3e3739aadaf2540b08abb8b2e77bd2eba57ee2a8843"}
Oct 30 19:45:41.527779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.527757    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:41.529854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.529827    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5" event={"ID":"5ed38bc4-b9f6-490b-a2dd-1977e88971bb","Type":"ContainerStarted","Data":"1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e"}
Oct 30 19:45:41.530003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.529869    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5" event={"ID":"5ed38bc4-b9f6-490b-a2dd-1977e88971bb","Type":"ContainerStarted","Data":"0a9e38a4d595e4264f265971bfcae37d4f55067b33355bc260f4de91dd9831f6"}
Oct 30 19:45:41.530003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.529903    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:41.575093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.549542    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z" podStartSLOduration=1.5495222659999999 podStartE2EDuration="1.549522266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:41.546823089 +0000 UTC m=+2323.543738473" watchObservedRunningTime="2026-10-30 19:45:41.549522266 +0000 UTC m=+2323.546437652"
Oct 30 19:45:41.596173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:41.596107    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5" podStartSLOduration=1.596086654 podStartE2EDuration="1.596086654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:41.589596506 +0000 UTC m=+2323.586511892" watchObservedRunningTime="2026-10-30 19:45:41.596086654 +0000 UTC m=+2323.593002040"
Oct 30 19:45:42.404083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:42.403070    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"]
Oct 30 19:45:42.404083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:42.403327    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy" containerID="cri-o://3ec8ee47cd8524db374f6f07f4fbc55c45d389c1fdb163a426893b4c38690811" gracePeriod=30
Oct 30 19:45:43.646231 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:45:43.646196    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:45:47.199766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.199719    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"]
Oct 30 19:45:47.212042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.212007    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:45:47.220692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.220645    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"]
Oct 30 19:45:47.222250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.221818    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c2b092\""
Oct 30 19:45:47.223784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.223613    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c2b092-dockercfg-5vjkr\""
Oct 30 19:45:47.251514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.250727    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"]
Oct 30 19:45:47.285114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.283630    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7558c\" (UniqueName: \"kubernetes.io/projected/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-kube-api-access-7558c\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5\" (UID: \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:45:47.285114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.283868    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5\" (UID: \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:45:47.385710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.385323    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7558c\" (UniqueName: \"kubernetes.io/projected/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-kube-api-access-7558c\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5\" (UID: \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:45:47.385710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.385540    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5\" (UID: \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:45:47.386616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.386585    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5\" (UID: \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:45:47.401756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.401709    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7558c\" (UniqueName: \"kubernetes.io/projected/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-kube-api-access-7558c\") pod \"payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5\" (UID: \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:45:47.603818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.575495    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:45:47.829083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.823181    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": read tcp 10.134.0.2:39696->10.134.0.254:15021: read: connection reset by peer" start-of-body=
Oct 30 19:45:47.829083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:47.823261    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": read tcp 10.134.0.2:39696->10.134.0.254:15021: read: connection reset by peer"
Oct 30 19:45:48.043153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.042592    6777 generic.go:356] "Generic (PLEG): container finished" podID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerID="3ec8ee47cd8524db374f6f07f4fbc55c45d389c1fdb163a426893b4c38690811" exitCode=0
Oct 30 19:45:48.043153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.042912    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" event={"ID":"8e937cd6-3dcd-4309-9494-3df69a56d551","Type":"ContainerDied","Data":"3ec8ee47cd8524db374f6f07f4fbc55c45d389c1fdb163a426893b4c38690811"}
Oct 30 19:45:48.043153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.042982    6777 scope.go:122] "RemoveContainer" containerID="0e3232805a6b390d0e668734d511446ffb2394c76d5dc6bb676bc99d3fdbc4d0"
Oct 30 19:45:48.048181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.046930    6777 generic.go:356] "Generic (PLEG): container finished" podID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerID="94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549" exitCode=0
Oct 30 19:45:48.048181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.047100    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" event={"ID":"85923a0f-3daa-4295-9a1d-3ca15689f158","Type":"ContainerDied","Data":"94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549"}
Oct 30 19:45:48.048181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.047690    6777 scope.go:122] "RemoveContainer" containerID="94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549"
Oct 30 19:45:48.052777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.050758    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:48.052777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.050857    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:48.052777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.050908    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:48.160344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.160276    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:45:48.277037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.276994    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-podinfo\") pod \"8e937cd6-3dcd-4309-9494-3df69a56d551\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") "
Oct 30 19:45:48.277037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.277049    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-certs\") pod \"8e937cd6-3dcd-4309-9494-3df69a56d551\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") "
Oct 30 19:45:48.277659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.277122    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-credential-socket\") pod \"8e937cd6-3dcd-4309-9494-3df69a56d551\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") "
Oct 30 19:45:48.277659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.277239    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-token\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-token\") pod \"8e937cd6-3dcd-4309-9494-3df69a56d551\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") "
Oct 30 19:45:48.277659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.277444    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e937cd6-3dcd-4309-9494-3df69a56d551-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e937cd6-3dcd-4309-9494-3df69a56d551-istiod-ca-cert\") pod \"8e937cd6-3dcd-4309-9494-3df69a56d551\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") "
Oct 30 19:45:48.277659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.277494    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-data\") pod \"8e937cd6-3dcd-4309-9494-3df69a56d551\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") "
Oct 30 19:45:48.277659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.277556    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-envoy\") pod \"8e937cd6-3dcd-4309-9494-3df69a56d551\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") "
Oct 30 19:45:48.277659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.277588    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-kube-api-access-8qn9f\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-kube-api-access-8qn9f\") pod \"8e937cd6-3dcd-4309-9494-3df69a56d551\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") "
Oct 30 19:45:48.278001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.277748    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-socket\") pod \"8e937cd6-3dcd-4309-9494-3df69a56d551\" (UID: \"8e937cd6-3dcd-4309-9494-3df69a56d551\") "
Oct 30 19:45:48.278969 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.278927    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-certs" pod "8e937cd6-3dcd-4309-9494-3df69a56d551" (UID: "8e937cd6-3dcd-4309-9494-3df69a56d551"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.279816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.279765    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-credential-socket" pod "8e937cd6-3dcd-4309-9494-3df69a56d551" (UID: "8e937cd6-3dcd-4309-9494-3df69a56d551"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.279816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.279796    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e937cd6-3dcd-4309-9494-3df69a56d551-istiod-ca-cert" pod "8e937cd6-3dcd-4309-9494-3df69a56d551" (UID: "8e937cd6-3dcd-4309-9494-3df69a56d551"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:48.280074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.280018    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-socket" pod "8e937cd6-3dcd-4309-9494-3df69a56d551" (UID: "8e937cd6-3dcd-4309-9494-3df69a56d551"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.280799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.280754    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-data" pod "8e937cd6-3dcd-4309-9494-3df69a56d551" (UID: "8e937cd6-3dcd-4309-9494-3df69a56d551"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.284654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.284611    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-token" pod "8e937cd6-3dcd-4309-9494-3df69a56d551" (UID: "8e937cd6-3dcd-4309-9494-3df69a56d551"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:48.286517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.286475    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-podinfo" pod "8e937cd6-3dcd-4309-9494-3df69a56d551" (UID: "8e937cd6-3dcd-4309-9494-3df69a56d551"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:48.286822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.286786    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-kube-api-access-8qn9f" pod "8e937cd6-3dcd-4309-9494-3df69a56d551" (UID: "8e937cd6-3dcd-4309-9494-3df69a56d551"). InnerVolumeSpecName "kube-api-access-8qn9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:48.287040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.287000    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-envoy" pod "8e937cd6-3dcd-4309-9494-3df69a56d551" (UID: "8e937cd6-3dcd-4309-9494-3df69a56d551"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.381362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.381319    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:48.381362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.381355    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qn9f\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-kube-api-access-8qn9f\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:48.381602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.381370    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:48.381602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.381384    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:48.381602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.381396    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:48.381602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.381408    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:48.381602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.381420    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:48.381602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.381432    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e937cd6-3dcd-4309-9494-3df69a56d551-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:48.381602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.381445    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e937cd6-3dcd-4309-9494-3df69a56d551-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:48.389905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:48.389873    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"]
Oct 30 19:45:48.392405 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:48.392374    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae58dd48_af4d_4fc9_b0ce_3ef47507e4b0.slice/crio-c8274e0296a9f24bbd12506a07696073573baee20acafb7230553889f56916a0 WatchSource:0}: Error finding container c8274e0296a9f24bbd12506a07696073573baee20acafb7230553889f56916a0: Status 404 returned error can't find the container with id c8274e0296a9f24bbd12506a07696073573baee20acafb7230553889f56916a0
Oct 30 19:45:49.088425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.087438    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5" event={"ID":"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0","Type":"ContainerStarted","Data":"cb9c10636a896956bc174f9e399e052765a51fa2f2e248a28f9406632c64e8b4"}
Oct 30 19:45:49.088425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.087506    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:45:49.088425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.087526    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5" event={"ID":"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0","Type":"ContainerStarted","Data":"c8274e0296a9f24bbd12506a07696073573baee20acafb7230553889f56916a0"}
Oct 30 19:45:49.097091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.096492    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" event={"ID":"85923a0f-3daa-4295-9a1d-3ca15689f158","Type":"ContainerStarted","Data":"8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af"}
Oct 30 19:45:49.109989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.105761    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98" event={"ID":"8e937cd6-3dcd-4309-9494-3df69a56d551","Type":"ContainerDied","Data":"bee3e96be5de4b7c4516baddf067d644e67e674132703b6b6394d16622db7afb"}
Oct 30 19:45:49.109989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.105823    6777 scope.go:122] "RemoveContainer" containerID="3ec8ee47cd8524db374f6f07f4fbc55c45d389c1fdb163a426893b4c38690811"
Oct 30 19:45:49.109989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.105950    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"
Oct 30 19:45:49.121473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.121337    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5" podStartSLOduration=2.121320556 podStartE2EDuration="2.121320556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:49.116507576 +0000 UTC m=+2331.113422961" watchObservedRunningTime="2026-10-30 19:45:49.121320556 +0000 UTC m=+2331.118235984"
Oct 30 19:45:49.198857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.198775    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"]
Oct 30 19:45:49.207331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.207278    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f5613a-openshift-default-55d9bc5bff-wcb98"]
Oct 30 19:45:49.701517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:49.701479    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"]
Oct 30 19:45:50.092082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:50.084494    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:45:50.092082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:50.084642    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 30 19:45:50.092082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:50.084691    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 30 19:45:50.452140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:50.451552    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" path="/var/lib/kubelet/pods/8e937cd6-3dcd-4309-9494-3df69a56d551/volumes"
Oct 30 19:45:51.084642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.083535    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 30 19:45:51.084642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.083598    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 30 19:45:51.190760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.190720    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"]
Oct 30 19:45:51.192009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.191928    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy"
Oct 30 19:45:51.192190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.192175    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy"
Oct 30 19:45:51.192272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.192265    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy"
Oct 30 19:45:51.192330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.192323    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy"
Oct 30 19:45:51.192552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.192543    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy"
Oct 30 19:45:51.192622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.192615    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e937cd6-3dcd-4309-9494-3df69a56d551" containerName="istio-proxy"
Oct 30 19:45:51.201143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.200117    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"]
Oct 30 19:45:51.201143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.200251    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:45:51.205266 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.205232    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"]
Oct 30 19:45:51.208616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.208528    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aa49c5-dockercfg-6w8dw\""
Oct 30 19:45:51.208616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.208590    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa49c5\""
Oct 30 19:45:51.218029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.217985    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f21a64-764f-45fc-bc16-bc7619cf1694-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th\" (UID: \"82f21a64-764f-45fc-bc16-bc7619cf1694\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:45:51.218290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.218267    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflh5\" (UniqueName: \"kubernetes.io/projected/82f21a64-764f-45fc-bc16-bc7619cf1694-kube-api-access-bflh5\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th\" (UID: \"82f21a64-764f-45fc-bc16-bc7619cf1694\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:45:51.235921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.233326    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"]
Oct 30 19:45:51.235921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.233440    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:45:51.582306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.582267    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f21a64-764f-45fc-bc16-bc7619cf1694-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th\" (UID: \"82f21a64-764f-45fc-bc16-bc7619cf1694\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:45:51.582751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.582720    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88579cc7-45cc-441d-bd4e-15cf9cacc928-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx\" (UID: \"88579cc7-45cc-441d-bd4e-15cf9cacc928\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:45:51.583173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.583146    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsddt\" (UniqueName: \"kubernetes.io/projected/88579cc7-45cc-441d-bd4e-15cf9cacc928-kube-api-access-fsddt\") pod \"payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx\" (UID: \"88579cc7-45cc-441d-bd4e-15cf9cacc928\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:45:51.583363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.583346    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflh5\" (UniqueName: \"kubernetes.io/projected/82f21a64-764f-45fc-bc16-bc7619cf1694-kube-api-access-bflh5\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th\" (UID: \"82f21a64-764f-45fc-bc16-bc7619cf1694\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:45:51.601626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.595963    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f21a64-764f-45fc-bc16-bc7619cf1694-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th\" (UID: \"82f21a64-764f-45fc-bc16-bc7619cf1694\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:45:51.632087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.627562    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:45:51.772554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.738202    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88579cc7-45cc-441d-bd4e-15cf9cacc928-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx\" (UID: \"88579cc7-45cc-441d-bd4e-15cf9cacc928\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:45:51.772554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.738390    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsddt\" (UniqueName: \"kubernetes.io/projected/88579cc7-45cc-441d-bd4e-15cf9cacc928-kube-api-access-fsddt\") pod \"payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx\" (UID: \"88579cc7-45cc-441d-bd4e-15cf9cacc928\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:45:51.799772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.784237    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88579cc7-45cc-441d-bd4e-15cf9cacc928-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx\" (UID: \"88579cc7-45cc-441d-bd4e-15cf9cacc928\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:45:51.951166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:51.950725    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflh5\" (UniqueName: \"kubernetes.io/projected/82f21a64-764f-45fc-bc16-bc7619cf1694-kube-api-access-bflh5\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th\" (UID: \"82f21a64-764f-45fc-bc16-bc7619cf1694\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:45:52.015100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:52.008991    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsddt\" (UniqueName: \"kubernetes.io/projected/88579cc7-45cc-441d-bd4e-15cf9cacc928-kube-api-access-fsddt\") pod \"payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx\" (UID: \"88579cc7-45cc-441d-bd4e-15cf9cacc928\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:45:52.083106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:52.080806    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:45:52.107084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:52.087887    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 30 19:45:52.107084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:52.087949    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 30 19:45:52.149179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:52.148143    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:45:52.252210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:52.252153    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2" podUID="1518a909-687d-4d36-bdc7-58a016b1e661" containerName="payload-processing" containerID="cri-o://f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397" gracePeriod=25
Oct 30 19:45:52.908339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:52.908281    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:45:53.004410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.004322    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"]
Oct 30 19:45:53.011993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.011957    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"]
Oct 30 19:45:53.089264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.083489    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 30 19:45:53.089264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.083556    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 30 19:45:53.283596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.283549    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2" podUID="1518a909-687d-4d36-bdc7-58a016b1e661" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:53.430248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.430211    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:45:53.430488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.430265    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:53.619544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.619444    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx" event={"ID":"88579cc7-45cc-441d-bd4e-15cf9cacc928","Type":"ContainerStarted","Data":"05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153"}
Oct 30 19:45:53.620473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.619818    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx" event={"ID":"88579cc7-45cc-441d-bd4e-15cf9cacc928","Type":"ContainerStarted","Data":"6f78bf6b51d2ae232a0966d76e7d718015761ef24b5958fcfe370c93813032b3"}
Oct 30 19:45:53.621165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.621104    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:45:53.623656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.623627    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th" event={"ID":"82f21a64-764f-45fc-bc16-bc7619cf1694","Type":"ContainerStarted","Data":"0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92"}
Oct 30 19:45:53.623773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.623683    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th" event={"ID":"82f21a64-764f-45fc-bc16-bc7619cf1694","Type":"ContainerStarted","Data":"644b9d95ac8fe4c62bc05b10bdd720cf0d976c36026fc6ee62482d827330186d"}
Oct 30 19:45:53.623851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.623816    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:45:53.642359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.642297    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx" podStartSLOduration=2.642279479 podStartE2EDuration="2.642279479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:53.639676655 +0000 UTC m=+2335.636592043" watchObservedRunningTime="2026-10-30 19:45:53.642279479 +0000 UTC m=+2335.639194924"
Oct 30 19:45:53.680108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.675216    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th" podStartSLOduration=2.675195655 podStartE2EDuration="2.675195655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:53.672565647 +0000 UTC m=+2335.669481032" watchObservedRunningTime="2026-10-30 19:45:53.675195655 +0000 UTC m=+2335.672111041"
Oct 30 19:45:53.921033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:53.921004    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:45:54.111855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:54.080639    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 30 19:45:54.125111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:54.120197    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 30 19:45:54.702240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:54.702186    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb" podUID="bf77670b-ea82-4af3-98a8-c112442539a3" containerName="payload-processing" containerID="cri-o://b52c60e70c0bae9e275d98eed38249d080ce96a3078b71c90c72ebd93809e182" gracePeriod=25
Oct 30 19:45:54.922091 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:54.919869    6777 logging.go:55] [core] [Channel #7255 SubChannel #7256]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.12:50052", ServerName: "10.134.1.12:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.12:50052: connect: connection refused"
Oct 30 19:45:55.082366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:55.082329    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 30 19:45:55.082677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:55.082644    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 30 19:45:55.934733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:55.934535    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb" podUID="bf77670b-ea82-4af3-98a8-c112442539a3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.12:50052\" within 1s: context deadline exceeded"
Oct 30 19:45:56.066305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.065973    6777 generic.go:356] "Generic (PLEG): container finished" podID="bf77670b-ea82-4af3-98a8-c112442539a3" containerID="b52c60e70c0bae9e275d98eed38249d080ce96a3078b71c90c72ebd93809e182" exitCode=0
Oct 30 19:45:56.067878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.067846    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb" event={"ID":"bf77670b-ea82-4af3-98a8-c112442539a3","Type":"ContainerDied","Data":"b52c60e70c0bae9e275d98eed38249d080ce96a3078b71c90c72ebd93809e182"}
Oct 30 19:45:56.085136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.084363    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 30 19:45:56.085136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.084432    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 30 19:45:56.209276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.209242    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:56.305177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.279780    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf77670b-ea82-4af3-98a8-c112442539a3-kube-api-access-zklml\" (UniqueName: \"kubernetes.io/projected/bf77670b-ea82-4af3-98a8-c112442539a3-kube-api-access-zklml\") pod \"bf77670b-ea82-4af3-98a8-c112442539a3\" (UID: \"bf77670b-ea82-4af3-98a8-c112442539a3\") "
Oct 30 19:45:56.305177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.279998    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf77670b-ea82-4af3-98a8-c112442539a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf77670b-ea82-4af3-98a8-c112442539a3-plugins-config-volume\") pod \"bf77670b-ea82-4af3-98a8-c112442539a3\" (UID: \"bf77670b-ea82-4af3-98a8-c112442539a3\") "
Oct 30 19:45:56.305177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.282994    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf77670b-ea82-4af3-98a8-c112442539a3-plugins-config-volume" pod "bf77670b-ea82-4af3-98a8-c112442539a3" (UID: "bf77670b-ea82-4af3-98a8-c112442539a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:56.323426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.323277    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf77670b-ea82-4af3-98a8-c112442539a3-kube-api-access-zklml" pod "bf77670b-ea82-4af3-98a8-c112442539a3" (UID: "bf77670b-ea82-4af3-98a8-c112442539a3"). InnerVolumeSpecName "kube-api-access-zklml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:56.386183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.385630    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zklml\" (UniqueName: \"kubernetes.io/projected/bf77670b-ea82-4af3-98a8-c112442539a3-kube-api-access-zklml\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:56.386183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:56.385676    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf77670b-ea82-4af3-98a8-c112442539a3-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:57.086794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:57.086760    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:45:57.087373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:57.087353    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:45:57.088273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:57.088238    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:45:57.089465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:57.089432    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb" event={"ID":"bf77670b-ea82-4af3-98a8-c112442539a3","Type":"ContainerDied","Data":"51bdd9a24adbc318bde56a26dc62c04068607b63581e23514cb69e7f632d15e4"}
Oct 30 19:45:57.089561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:57.089499    6777 scope.go:122] "RemoveContainer" containerID="b52c60e70c0bae9e275d98eed38249d080ce96a3078b71c90c72ebd93809e182"
Oct 30 19:45:57.089710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:57.089693    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"
Oct 30 19:45:57.134180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:57.134146    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"]
Oct 30 19:45:57.141107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:57.141070    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-56b65445b4-sf4rb"]
Oct 30 19:45:58.101565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.101529    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"]
Oct 30 19:45:58.102772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.102735    6777 cpu_manager.go:414] "containerMap: removing container" podUID="bf77670b-ea82-4af3-98a8-c112442539a3" containerName="payload-processing"
Oct 30 19:45:58.102772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.102754    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf77670b-ea82-4af3-98a8-c112442539a3" containerName="payload-processing"
Oct 30 19:45:58.103037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.103023    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf77670b-ea82-4af3-98a8-c112442539a3" containerName="payload-processing"
Oct 30 19:45:58.113445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.113410    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.190346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.190307    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf77670b-ea82-4af3-98a8-c112442539a3" path="/var/lib/kubelet/pods/bf77670b-ea82-4af3-98a8-c112442539a3/volumes"
Oct 30 19:45:58.190763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.190742    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"]
Oct 30 19:45:58.221286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.221245    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.221498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.221311    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.221498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.221386    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.221498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.221434    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed339902-7cac-4388-b69e-f1768cfe44e0-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.221498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.221463    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69twq\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-kube-api-access-69twq\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.221670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.221646    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.223522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.221710    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.223522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.221788    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.223522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.221883    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.323463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.323419    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed339902-7cac-4388-b69e-f1768cfe44e0-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.323463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.323469    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69twq\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-kube-api-access-69twq\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.323929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.323602    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.323929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.323657    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.323929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.323700    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.323929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.323759    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.323929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.323916    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.324276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.323963    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.324276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.324024    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.324276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.324151    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.324276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.324171    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.324504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.324387    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.324504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.324416    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed339902-7cac-4388-b69e-f1768cfe44e0-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.324504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.324427    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.326154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.326135    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.326393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.326376    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.339853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.339820    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.340188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.340158    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69twq\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-kube-api-access-69twq\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.480894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.480801    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:45:58.780846 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:45:58.780814    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded339902_7cac_4388_b69e_f1768cfe44e0.slice/crio-a4689204adbf35c59b4fa0d4037b8e979d5ddcf85866556fcace1e97379e69d9 WatchSource:0}: Error finding container a4689204adbf35c59b4fa0d4037b8e979d5ddcf85866556fcace1e97379e69d9: Status 404 returned error can't find the container with id a4689204adbf35c59b4fa0d4037b8e979d5ddcf85866556fcace1e97379e69d9
Oct 30 19:45:58.780987 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.780847    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"]
Oct 30 19:45:58.783961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.783928    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:58.784047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.783996    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:58.784047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:58.784026    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:59.112795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.112760    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" event={"ID":"ed339902-7cac-4388-b69e-f1768cfe44e0","Type":"ContainerStarted","Data":"387fb8a0f1959c3f51aa6b0defa3fcb86ba01187d59b2f4ea0d7726b4eddaffb"}
Oct 30 19:45:59.113378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.112822    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" event={"ID":"ed339902-7cac-4388-b69e-f1768cfe44e0","Type":"ContainerStarted","Data":"a4689204adbf35c59b4fa0d4037b8e979d5ddcf85866556fcace1e97379e69d9"}
Oct 30 19:45:59.114756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.114707    6777 generic.go:356] "Generic (PLEG): container finished" podID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerID="206c54088f0a9d30ba7d4a55053462985cacc6ab41e5035ae4175334ad9b16e7" exitCode=0
Oct 30 19:45:59.114906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.114781    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" event={"ID":"fcfd0a26-981e-4313-b6d4-66dbf0107982","Type":"ContainerDied","Data":"206c54088f0a9d30ba7d4a55053462985cacc6ab41e5035ae4175334ad9b16e7"}
Oct 30 19:45:59.115545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.115522    6777 scope.go:122] "RemoveContainer" containerID="206c54088f0a9d30ba7d4a55053462985cacc6ab41e5035ae4175334ad9b16e7"
Oct 30 19:45:59.117742 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.117715    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:59.117841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.117784    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:59.117841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.117817    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:45:59.138403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.138346    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" podStartSLOduration=1.138328859 podStartE2EDuration="1.138328859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:59.136160446 +0000 UTC m=+2341.133075832" watchObservedRunningTime="2026-10-30 19:45:59.138328859 +0000 UTC m=+2341.135244242"
Oct 30 19:45:59.491202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.485182    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Oct 30 19:45:59.491202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.485246    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Oct 30 19:45:59.491202 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:45:59.485561    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:46:00.127048 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:00.126945    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" event={"ID":"fcfd0a26-981e-4313-b6d4-66dbf0107982","Type":"ContainerStarted","Data":"9fa629c139054aa591fa11dcdf878f358253cf52483fa9133d19dd9105eb2738"}
Oct 30 19:46:00.433162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:00.432505    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:46:00.503113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:00.501709    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Oct 30 19:46:00.503113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:00.501776    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Oct 30 19:46:00.853801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:00.853760    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:46:00.859652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:00.859518    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:46:00.860180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:00.860046    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 19:46:01.495085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:01.491684    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Oct 30 19:46:01.495085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:01.491747    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Oct 30 19:46:02.022514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:01.864744    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:46:02.022514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:01.864814    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 19:46:02.485367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:02.485324    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Oct 30 19:46:02.485691 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:02.485656    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Oct 30 19:46:02.850811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:02.850765    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:46:02.853273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:02.853215    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 19:46:03.073294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:03.073248    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2" podUID="1518a909-687d-4d36-bdc7-58a016b1e661" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:46:03.496085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:03.492539    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Oct 30 19:46:03.496085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:03.492611    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Oct 30 19:46:03.848993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:03.848956    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:46:03.867501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:03.867453    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 19:46:04.322298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:04.322234    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"]
Oct 30 19:46:04.322944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:04.322896    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy" containerID="cri-o://7e08bc99cef45be329c48e90944cf9ff4f72942ab3942515a46b63daee12700b" gracePeriod=30
Oct 30 19:46:04.531231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:04.518197    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Oct 30 19:46:04.531231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:04.518285    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Oct 30 19:46:04.666962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:04.666520    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:46:04.928159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:04.914337    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:46:04.928159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:04.914407    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 19:46:05.177235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:05.171989    6777 scope.go:122] "RemoveContainer" containerID="4f4932abe864d1b1a7b574ce4cc0c2e38231f884f08c5b442345838d83e2bb71"
Oct 30 19:46:05.220921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:05.220890    6777 scope.go:122] "RemoveContainer" containerID="7c46a70d832f8563b660f939024f0919a17500d53ad559cc45d960402777c297"
Oct 30 19:46:05.236567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:05.236464    6777 scope.go:122] "RemoveContainer" containerID="9911ae24c93010e7e2c64798a8765241497a7d037feefd6d3fa6012c469d7878"
Oct 30 19:46:05.343867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:05.343833    6777 scope.go:122] "RemoveContainer" containerID="cc22aa89829045f5d382f6c5705243bee7662f4d18d786d2c4c5778da7318093"
Oct 30 19:46:05.438531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:05.438501    6777 scope.go:122] "RemoveContainer" containerID="1ce484c0a3f329f49e10ae4c3b32948229d15c89179a9ca0d55e29d8603388aa"
Oct 30 19:46:05.487759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:05.487657    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Oct 30 19:46:05.488126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:05.488089    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Oct 30 19:46:05.881151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:05.867454    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:46:05.881151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:05.867520    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 19:46:06.102321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:06.102279    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:46:06.611148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:06.607412    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:46:06.611148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:06.609219    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:46:06.612045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:06.611918    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:46:06.860958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:06.860923    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:46:06.861483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:06.861409    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:46:06.862470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:06.862451    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:46:06.935958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:06.935920    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:06.936163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:06.936004    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:08.516522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:08.516479    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:46:08.517675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:08.517221    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:46:09.571897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.571852    6777 generic.go:356] "Generic (PLEG): container finished" podID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerID="7e08bc99cef45be329c48e90944cf9ff4f72942ab3942515a46b63daee12700b" exitCode=0
Oct 30 19:46:09.572391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.571918    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" event={"ID":"09659fd9-9629-41e5-9e07-e6680dfa05e4","Type":"ContainerDied","Data":"7e08bc99cef45be329c48e90944cf9ff4f72942ab3942515a46b63daee12700b"}
Oct 30 19:46:09.572391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.571975    6777 scope.go:122] "RemoveContainer" containerID="f1315dabf78f49aea8392827fd827a5e7722eaeb2bffbca7114380878fba13ef"
Oct 30 19:46:09.723344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.723295    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:46:09.825371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.825327    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"]
Oct 30 19:46:09.831500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.831463    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-podinfo\") pod \"09659fd9-9629-41e5-9e07-e6680dfa05e4\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") "
Oct 30 19:46:09.831936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.831912    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09659fd9-9629-41e5-9e07-e6680dfa05e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09659fd9-9629-41e5-9e07-e6680dfa05e4-istiod-ca-cert\") pod \"09659fd9-9629-41e5-9e07-e6680dfa05e4\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") "
Oct 30 19:46:09.832492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.832451    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09659fd9-9629-41e5-9e07-e6680dfa05e4-istiod-ca-cert" pod "09659fd9-9629-41e5-9e07-e6680dfa05e4" (UID: "09659fd9-9629-41e5-9e07-e6680dfa05e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:09.832674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.832652    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-credential-socket\") pod \"09659fd9-9629-41e5-9e07-e6680dfa05e4\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") "
Oct 30 19:46:09.832784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.832759    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-envoy\") pod \"09659fd9-9629-41e5-9e07-e6680dfa05e4\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") "
Oct 30 19:46:09.832857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.832799    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-certs\") pod \"09659fd9-9629-41e5-9e07-e6680dfa05e4\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") "
Oct 30 19:46:09.832857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.832826    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-socket\") pod \"09659fd9-9629-41e5-9e07-e6680dfa05e4\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") "
Oct 30 19:46:09.832966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.832866    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-token\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-token\") pod \"09659fd9-9629-41e5-9e07-e6680dfa05e4\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") "
Oct 30 19:46:09.832966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.832900    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-kube-api-access-vjwgx\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-kube-api-access-vjwgx\") pod \"09659fd9-9629-41e5-9e07-e6680dfa05e4\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") "
Oct 30 19:46:09.833136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.832995    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-data\") pod \"09659fd9-9629-41e5-9e07-e6680dfa05e4\" (UID: \"09659fd9-9629-41e5-9e07-e6680dfa05e4\") "
Oct 30 19:46:09.833136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.833015    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-credential-socket" pod "09659fd9-9629-41e5-9e07-e6680dfa05e4" (UID: "09659fd9-9629-41e5-9e07-e6680dfa05e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.833243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.833220    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-certs" pod "09659fd9-9629-41e5-9e07-e6680dfa05e4" (UID: "09659fd9-9629-41e5-9e07-e6680dfa05e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.833672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.833644    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-data" pod "09659fd9-9629-41e5-9e07-e6680dfa05e4" (UID: "09659fd9-9629-41e5-9e07-e6680dfa05e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.833866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.833842    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-socket" pod "09659fd9-9629-41e5-9e07-e6680dfa05e4" (UID: "09659fd9-9629-41e5-9e07-e6680dfa05e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.834236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.834215    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.834319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.834243    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.834319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.834259    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.834319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.834274    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.834319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.834287    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09659fd9-9629-41e5-9e07-e6680dfa05e4-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.835214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.835192    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-podinfo" pod "09659fd9-9629-41e5-9e07-e6680dfa05e4" (UID: "09659fd9-9629-41e5-9e07-e6680dfa05e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:09.835330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.835312    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-envoy" pod "09659fd9-9629-41e5-9e07-e6680dfa05e4" (UID: "09659fd9-9629-41e5-9e07-e6680dfa05e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.835727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.835702    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-token" pod "09659fd9-9629-41e5-9e07-e6680dfa05e4" (UID: "09659fd9-9629-41e5-9e07-e6680dfa05e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:09.835905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.835889    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-kube-api-access-vjwgx" pod "09659fd9-9629-41e5-9e07-e6680dfa05e4" (UID: "09659fd9-9629-41e5-9e07-e6680dfa05e4"). InnerVolumeSpecName "kube-api-access-vjwgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:09.935953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.935839    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.935953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.935891    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.935953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.935904    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjwgx\" (UniqueName: \"kubernetes.io/projected/09659fd9-9629-41e5-9e07-e6680dfa05e4-kube-api-access-vjwgx\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.935953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:09.935913    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09659fd9-9629-41e5-9e07-e6680dfa05e4-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.188350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.188242    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"]
Oct 30 19:46:10.189225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.189199    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy"
Oct 30 19:46:10.189225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.189225    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy"
Oct 30 19:46:10.189509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.189494    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy"
Oct 30 19:46:10.189574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.189516    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy"
Oct 30 19:46:10.193944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.193915    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:10.202114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.198923    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-826518-dockercfg-kqlq2\""
Oct 30 19:46:10.203285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.203255    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"]
Oct 30 19:46:10.242201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.242162    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-826518\""
Oct 30 19:46:10.509153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.505053    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf\" (UID: \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:10.509153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.505161    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8vq\" (UniqueName: \"kubernetes.io/projected/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-kube-api-access-6z8vq\") pod \"payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf\" (UID: \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:10.582593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.582559    6777 generic.go:356] "Generic (PLEG): container finished" podID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerID="87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487" exitCode=0
Oct 30 19:46:10.583031 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.582628    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" event={"ID":"58823786-a7bd-4cc8-9a20-da28e6921be2","Type":"ContainerDied","Data":"87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487"}
Oct 30 19:46:10.583365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.583345    6777 scope.go:122] "RemoveContainer" containerID="87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487"
Oct 30 19:46:10.584471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.584426    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt" event={"ID":"09659fd9-9629-41e5-9e07-e6680dfa05e4","Type":"ContainerDied","Data":"3ba7cc4de3b682ad41cdf447d34c61b68661650ed41c8389ee1cc5540c9ecfc7"}
Oct 30 19:46:10.584471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.584465    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"
Oct 30 19:46:10.584620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.584483    6777 scope.go:122] "RemoveContainer" containerID="7e08bc99cef45be329c48e90944cf9ff4f72942ab3942515a46b63daee12700b"
Oct 30 19:46:10.585606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.585573    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:10.585684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.585641    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:10.585684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.585673    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:10.608870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.608831    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf\" (UID: \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:10.608870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.608883    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8vq\" (UniqueName: \"kubernetes.io/projected/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-kube-api-access-6z8vq\") pod \"payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf\" (UID: \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:10.609937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.609904    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf\" (UID: \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:10.634072 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.634023    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"]
Oct 30 19:46:10.634789 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.634753    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8vq\" (UniqueName: \"kubernetes.io/projected/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-kube-api-access-6z8vq\") pod \"payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf\" (UID: \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:10.643653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.643599    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-m7xzt"]
Oct 30 19:46:10.813418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:10.813378    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:11.158864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:11.158823    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"]
Oct 30 19:46:11.159574 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:46:11.159544    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51d41eb_ec1d_4a77_9b41_a4bd92c26a38.slice/crio-524faa13c63703f7df4ae1c977c324b92f023255bce091526b64601ac513a7bb WatchSource:0}: Error finding container 524faa13c63703f7df4ae1c977c324b92f023255bce091526b64601ac513a7bb: Status 404 returned error can't find the container with id 524faa13c63703f7df4ae1c977c324b92f023255bce091526b64601ac513a7bb
Oct 30 19:46:11.647207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:11.646409    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" event={"ID":"58823786-a7bd-4cc8-9a20-da28e6921be2","Type":"ContainerStarted","Data":"118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c"}
Oct 30 19:46:11.657544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:11.652899    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf" event={"ID":"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38","Type":"ContainerStarted","Data":"f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138"}
Oct 30 19:46:11.657544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:11.652968    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf" event={"ID":"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38","Type":"ContainerStarted","Data":"524faa13c63703f7df4ae1c977c324b92f023255bce091526b64601ac513a7bb"}
Oct 30 19:46:11.657544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:11.654034    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:11.708684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:11.708618    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf" podStartSLOduration=1.7085983059999998 podStartE2EDuration="1.708598306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:11.706504849 +0000 UTC m=+2353.703420235" watchObservedRunningTime="2026-10-30 19:46:11.708598306 +0000 UTC m=+2353.705513690"
Oct 30 19:46:12.361088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.352081    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" path="/var/lib/kubelet/pods/09659fd9-9629-41e5-9e07-e6680dfa05e4/volumes"
Oct 30 19:46:12.512582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.512537    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:46:12.512989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.512803    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:46:12.513155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.513022    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 19:46:12.633815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.633776    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:46:12.642101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.642045    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1518a909-687d-4d36-bdc7-58a016b1e661-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518a909-687d-4d36-bdc7-58a016b1e661-plugins-config-volume\") pod \"1518a909-687d-4d36-bdc7-58a016b1e661\" (UID: \"1518a909-687d-4d36-bdc7-58a016b1e661\") "
Oct 30 19:46:12.642293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.642198    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1518a909-687d-4d36-bdc7-58a016b1e661-kube-api-access-64b8q\" (UniqueName: \"kubernetes.io/projected/1518a909-687d-4d36-bdc7-58a016b1e661-kube-api-access-64b8q\") pod \"1518a909-687d-4d36-bdc7-58a016b1e661\" (UID: \"1518a909-687d-4d36-bdc7-58a016b1e661\") "
Oct 30 19:46:12.642471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.642438    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1518a909-687d-4d36-bdc7-58a016b1e661-plugins-config-volume" pod "1518a909-687d-4d36-bdc7-58a016b1e661" (UID: "1518a909-687d-4d36-bdc7-58a016b1e661"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:12.642826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.642804    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518a909-687d-4d36-bdc7-58a016b1e661-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:12.644644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.644613    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1518a909-687d-4d36-bdc7-58a016b1e661-kube-api-access-64b8q" pod "1518a909-687d-4d36-bdc7-58a016b1e661" (UID: "1518a909-687d-4d36-bdc7-58a016b1e661"). InnerVolumeSpecName "kube-api-access-64b8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:12.702798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.702751    6777 generic.go:356] "Generic (PLEG): container finished" podID="1518a909-687d-4d36-bdc7-58a016b1e661" containerID="f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397" exitCode=0
Oct 30 19:46:12.703382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.702862    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"
Oct 30 19:46:12.703382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.702899    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2" event={"ID":"1518a909-687d-4d36-bdc7-58a016b1e661","Type":"ContainerDied","Data":"f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397"}
Oct 30 19:46:12.703382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.702966    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2" event={"ID":"1518a909-687d-4d36-bdc7-58a016b1e661","Type":"ContainerDied","Data":"3a5f26e50b545d7b4ab766e5c9a4f553ee288cf050c4827a2d1df427f22a307d"}
Oct 30 19:46:12.703382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.702996    6777 scope.go:122] "RemoveContainer" containerID="f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397"
Oct 30 19:46:12.716554 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.716524    6777 scope.go:122] "RemoveContainer" containerID="f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397"
Oct 30 19:46:12.716938 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:12.716911    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397\": container with ID starting with f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397 not found: ID does not exist" containerID="f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397"
Oct 30 19:46:12.717026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.716954    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397"} err="failed to get container status \"f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397\": rpc error: code = NotFound desc = could not find container \"f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397\": container with ID starting with f2e1c2700ab61590936802063377bb663fa0c951cdffbe742606840487603397 not found: ID does not exist"
Oct 30 19:46:12.735570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.735534    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"]
Oct 30 19:46:12.742081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.742038    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e3586f-77459df5d6-vq2z2"]
Oct 30 19:46:12.745142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:12.745113    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64b8q\" (UniqueName: \"kubernetes.io/projected/1518a909-687d-4d36-bdc7-58a016b1e661-kube-api-access-64b8q\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:13.513437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:13.513388    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:46:13.513646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:13.513463    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 19:46:13.643582 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:13.643544    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:46:14.117569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:14.117531    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1518a909-687d-4d36-bdc7-58a016b1e661" path="/var/lib/kubelet/pods/1518a909-687d-4d36-bdc7-58a016b1e661/volumes"
Oct 30 19:46:14.512680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:14.512583    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:46:14.512680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:14.512651    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 19:46:15.520098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:15.518892    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:46:15.520098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:15.518968    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 19:46:16.513718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:16.513676    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:46:16.513921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:16.513734    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 19:46:17.554466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:17.554433    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:46:17.554951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:17.554934    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:46:17.556393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:17.556371    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:46:19.752889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:19.752848    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"]
Oct 30 19:46:19.754613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:19.753146    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" containerID="cri-o://95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678" gracePeriod=30
Oct 30 19:46:19.760163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:19.760120    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"]
Oct 30 19:46:19.760433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:19.760404    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy" containerID="cri-o://1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd" gracePeriod=30
Oct 30 19:46:23.714732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:23.714696    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:46:23.809547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:23.809494    6777 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:23.809757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:23.809576    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:24.897968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:24.897877    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" containerID="cri-o://4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c" gracePeriod=45
Oct 30 19:46:25.172520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.172493    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:46:25.177328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.177304    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:46:25.232855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.232646    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-certs" pod "adb479ae-22d3-45b4-99d4-d9377e881735" (UID: "adb479ae-22d3-45b4-99d4-d9377e881735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.235230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.230497    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-certs\") pod \"adb479ae-22d3-45b4-99d4-d9377e881735\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") "
Oct 30 19:46:25.235500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.235478    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adb479ae-22d3-45b4-99d4-d9377e881735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb479ae-22d3-45b4-99d4-d9377e881735-istio-podinfo\") pod \"adb479ae-22d3-45b4-99d4-d9377e881735\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") "
Oct 30 19:46:25.235752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.235734    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-certs\") pod \"7e94e090-5bce-47cf-9c75-a776e08b3265\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") "
Oct 30 19:46:25.235902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.235886    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e94e090-5bce-47cf-9c75-a776e08b3265-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e94e090-5bce-47cf-9c75-a776e08b3265-istiod-ca-cert\") pod \"7e94e090-5bce-47cf-9c75-a776e08b3265\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") "
Oct 30 19:46:25.236049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.236032    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-credential-socket\") pod \"adb479ae-22d3-45b4-99d4-d9377e881735\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") "
Oct 30 19:46:25.236339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.236322    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-data\") pod \"adb479ae-22d3-45b4-99d4-d9377e881735\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") "
Oct 30 19:46:25.236508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.236494    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb479ae-22d3-45b4-99d4-d9377e881735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb479ae-22d3-45b4-99d4-d9377e881735-istiod-ca-cert\") pod \"adb479ae-22d3-45b4-99d4-d9377e881735\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") "
Oct 30 19:46:25.236617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.236605    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-token\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-token\") pod \"7e94e090-5bce-47cf-9c75-a776e08b3265\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") "
Oct 30 19:46:25.236950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.236932    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-istio-token\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-istio-token\") pod \"adb479ae-22d3-45b4-99d4-d9377e881735\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") "
Oct 30 19:46:25.237147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.237129    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-socket\") pod \"7e94e090-5bce-47cf-9c75-a776e08b3265\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") "
Oct 30 19:46:25.237398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.237381    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-envoy\") pod \"7e94e090-5bce-47cf-9c75-a776e08b3265\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") "
Oct 30 19:46:25.237559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.237545    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-socket\") pod \"adb479ae-22d3-45b4-99d4-d9377e881735\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") "
Oct 30 19:46:25.237671 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.237660    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-data\") pod \"7e94e090-5bce-47cf-9c75-a776e08b3265\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") "
Oct 30 19:46:25.237791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.237779    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-envoy\") pod \"adb479ae-22d3-45b4-99d4-d9377e881735\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") "
Oct 30 19:46:25.237916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.237904    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-credential-socket\") pod \"7e94e090-5bce-47cf-9c75-a776e08b3265\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") "
Oct 30 19:46:25.238026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.238012    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-kube-api-access-9d9zm\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-kube-api-access-9d9zm\") pod \"adb479ae-22d3-45b4-99d4-d9377e881735\" (UID: \"adb479ae-22d3-45b4-99d4-d9377e881735\") "
Oct 30 19:46:25.238918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.236492    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-certs" pod "7e94e090-5bce-47cf-9c75-a776e08b3265" (UID: "7e94e090-5bce-47cf-9c75-a776e08b3265"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.238918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.236671    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-credential-socket" pod "adb479ae-22d3-45b4-99d4-d9377e881735" (UID: "adb479ae-22d3-45b4-99d4-d9377e881735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.238918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.236969    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e94e090-5bce-47cf-9c75-a776e08b3265-istiod-ca-cert" pod "7e94e090-5bce-47cf-9c75-a776e08b3265" (UID: "7e94e090-5bce-47cf-9c75-a776e08b3265"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:25.238918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.237541    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb479ae-22d3-45b4-99d4-d9377e881735-istiod-ca-cert" pod "adb479ae-22d3-45b4-99d4-d9377e881735" (UID: "adb479ae-22d3-45b4-99d4-d9377e881735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:25.239529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.239259    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-socket" pod "adb479ae-22d3-45b4-99d4-d9377e881735" (UID: "adb479ae-22d3-45b4-99d4-d9377e881735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.239529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.239272    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-data" pod "adb479ae-22d3-45b4-99d4-d9377e881735" (UID: "adb479ae-22d3-45b4-99d4-d9377e881735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.239767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.239746    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-podinfo\") pod \"7e94e090-5bce-47cf-9c75-a776e08b3265\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") "
Oct 30 19:46:25.239934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.239918    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-kube-api-access-m2jft\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-kube-api-access-m2jft\") pod \"7e94e090-5bce-47cf-9c75-a776e08b3265\" (UID: \"7e94e090-5bce-47cf-9c75-a776e08b3265\") "
Oct 30 19:46:25.249578 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.240989    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb479ae-22d3-45b4-99d4-d9377e881735-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.254326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.254299    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.254507 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.254490    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.254692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.254677    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.254816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.254802    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e94e090-5bce-47cf-9c75-a776e08b3265-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.254927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.254915    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.255035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.255022    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.255175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.239926    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-data" pod "7e94e090-5bce-47cf-9c75-a776e08b3265" (UID: "7e94e090-5bce-47cf-9c75-a776e08b3265"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.255272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.240676    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-credential-socket" pod "7e94e090-5bce-47cf-9c75-a776e08b3265" (UID: "7e94e090-5bce-47cf-9c75-a776e08b3265"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.255367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.241982    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-envoy" pod "7e94e090-5bce-47cf-9c75-a776e08b3265" (UID: "7e94e090-5bce-47cf-9c75-a776e08b3265"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.255453 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.242682    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb479ae-22d3-45b4-99d4-d9377e881735-istio-podinfo" pod "adb479ae-22d3-45b4-99d4-d9377e881735" (UID: "adb479ae-22d3-45b4-99d4-d9377e881735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:25.255541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.243030    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-socket" pod "7e94e090-5bce-47cf-9c75-a776e08b3265" (UID: "7e94e090-5bce-47cf-9c75-a776e08b3265"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.255629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.248012    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-kube-api-access-m2jft" pod "7e94e090-5bce-47cf-9c75-a776e08b3265" (UID: "7e94e090-5bce-47cf-9c75-a776e08b3265"). InnerVolumeSpecName "kube-api-access-m2jft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:25.255724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.250272    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-istio-token" pod "adb479ae-22d3-45b4-99d4-d9377e881735" (UID: "adb479ae-22d3-45b4-99d4-d9377e881735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:25.266017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.265980    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-envoy" pod "adb479ae-22d3-45b4-99d4-d9377e881735" (UID: "adb479ae-22d3-45b4-99d4-d9377e881735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.266273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.266233    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-token" pod "7e94e090-5bce-47cf-9c75-a776e08b3265" (UID: "7e94e090-5bce-47cf-9c75-a776e08b3265"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:25.266529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.266508    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-kube-api-access-9d9zm" pod "adb479ae-22d3-45b4-99d4-d9377e881735" (UID: "adb479ae-22d3-45b4-99d4-d9377e881735"). InnerVolumeSpecName "kube-api-access-9d9zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:25.268559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.268535    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-podinfo" pod "7e94e090-5bce-47cf-9c75-a776e08b3265" (UID: "7e94e090-5bce-47cf-9c75-a776e08b3265"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:25.308396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.308368    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.356019    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-model-cache\") pod \"3c9e3c06-7423-46e4-b910-bba1a9147b70\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") "
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.356348    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-dshm\") pod \"3c9e3c06-7423-46e4-b910-bba1a9147b70\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") "
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.356391    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c9e3c06-7423-46e4-b910-bba1a9147b70-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e3c06-7423-46e4-b910-bba1a9147b70-tls-certs\") pod \"3c9e3c06-7423-46e4-b910-bba1a9147b70\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") "
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.356446    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-tmp-dir\") pod \"3c9e3c06-7423-46e4-b910-bba1a9147b70\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") "
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.356661    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9e3c06-7423-46e4-b910-bba1a9147b70-kube-api-access-66dkb\" (UniqueName: \"kubernetes.io/projected/3c9e3c06-7423-46e4-b910-bba1a9147b70-kube-api-access-66dkb\") pod \"3c9e3c06-7423-46e4-b910-bba1a9147b70\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") "
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.356711    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-home\") pod \"3c9e3c06-7423-46e4-b910-bba1a9147b70\" (UID: \"3c9e3c06-7423-46e4-b910-bba1a9147b70\") "
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.356972    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-model-cache" pod "3c9e3c06-7423-46e4-b910-bba1a9147b70" (UID: "3c9e3c06-7423-46e4-b910-bba1a9147b70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357217    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-home" pod "3c9e3c06-7423-46e4-b910-bba1a9147b70" (UID: "3c9e3c06-7423-46e4-b910-bba1a9147b70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.356982    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-tmp-dir" pod "3c9e3c06-7423-46e4-b910-bba1a9147b70" (UID: "3c9e3c06-7423-46e4-b910-bba1a9147b70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357655    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357677    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d9zm\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-kube-api-access-9d9zm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357691    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357705    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2jft\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-kube-api-access-m2jft\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357718    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb479ae-22d3-45b4-99d4-d9377e881735-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357730    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357741    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357755    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357766    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb479ae-22d3-45b4-99d4-d9377e881735-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357780    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357792    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357805    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e94e090-5bce-47cf-9c75-a776e08b3265-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357818    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb479ae-22d3-45b4-99d4-d9377e881735-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.358251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.357830    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.359941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.359912    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9e3c06-7423-46e4-b910-bba1a9147b70-tls-certs" pod "3c9e3c06-7423-46e4-b910-bba1a9147b70" (UID: "3c9e3c06-7423-46e4-b910-bba1a9147b70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:25.364674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.364563    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9e3c06-7423-46e4-b910-bba1a9147b70-kube-api-access-66dkb" pod "3c9e3c06-7423-46e4-b910-bba1a9147b70" (UID: "3c9e3c06-7423-46e4-b910-bba1a9147b70"). InnerVolumeSpecName "kube-api-access-66dkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:25.368558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.368528    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-dshm" pod "3c9e3c06-7423-46e4-b910-bba1a9147b70" (UID: "3c9e3c06-7423-46e4-b910-bba1a9147b70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.459808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.459773    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e3c06-7423-46e4-b910-bba1a9147b70-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.459808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.459804    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e3c06-7423-46e4-b910-bba1a9147b70-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.460035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.459816    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66dkb\" (UniqueName: \"kubernetes.io/projected/3c9e3c06-7423-46e4-b910-bba1a9147b70-kube-api-access-66dkb\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.883098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.872754    6777 generic.go:356] "Generic (PLEG): container finished" podID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerID="95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678" exitCode=0
Oct 30 19:46:25.883098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.872990    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" event={"ID":"7e94e090-5bce-47cf-9c75-a776e08b3265","Type":"ContainerDied","Data":"95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678"}
Oct 30 19:46:25.883098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.873043    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d" event={"ID":"7e94e090-5bce-47cf-9c75-a776e08b3265","Type":"ContainerDied","Data":"3238462c4e6015183c2b4ccd8ab8930c68bc170ebe4b9dcd1b14bfd8b520b4cd"}
Oct 30 19:46:25.883098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.873093    6777 scope.go:122] "RemoveContainer" containerID="95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678"
Oct 30 19:46:25.883098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.873265    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"
Oct 30 19:46:25.941365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.936091    6777 generic.go:356] "Generic (PLEG): container finished" podID="adb479ae-22d3-45b4-99d4-d9377e881735" containerID="1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd" exitCode=0
Oct 30 19:46:25.941365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.936322    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" event={"ID":"adb479ae-22d3-45b4-99d4-d9377e881735","Type":"ContainerDied","Data":"1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd"}
Oct 30 19:46:25.941365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.936373    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v" event={"ID":"adb479ae-22d3-45b4-99d4-d9377e881735","Type":"ContainerDied","Data":"09a556349f43660ce077dc970fbfa4e3482b06d95bf1548cc790795470ea0a33"}
Oct 30 19:46:25.941365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.936461    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"
Oct 30 19:46:25.953096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.942080    6777 generic.go:356] "Generic (PLEG): container finished" podID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerID="4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c" exitCode=0
Oct 30 19:46:25.953096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.942164    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" event={"ID":"3c9e3c06-7423-46e4-b910-bba1a9147b70","Type":"ContainerDied","Data":"4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c"}
Oct 30 19:46:25.953096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.942197    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" event={"ID":"3c9e3c06-7423-46e4-b910-bba1a9147b70","Type":"ContainerDied","Data":"2002d35d0b24ead4754f0f7692fc3c0560311d1b58e23e466758606eaa0b877f"}
Oct 30 19:46:25.953096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.942258    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"
Oct 30 19:46:25.984457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.984422    6777 scope.go:122] "RemoveContainer" containerID="95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678"
Oct 30 19:46:25.985704 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:25.985538    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678\": container with ID starting with 95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678 not found: ID does not exist" containerID="95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678"
Oct 30 19:46:25.985704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.985584    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678"} err="failed to get container status \"95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678\": rpc error: code = NotFound desc = could not find container \"95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678\": container with ID starting with 95c832878e2808e15b244d32a2a24f1e85029c0da865118ac9de29734985d678 not found: ID does not exist"
Oct 30 19:46:25.985704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:25.985612    6777 scope.go:122] "RemoveContainer" containerID="1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd"
Oct 30 19:46:26.013483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.013408    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"]
Oct 30 19:46:26.031735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.031702    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-vx26d"]
Oct 30 19:46:26.037780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.037740    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"]
Oct 30 19:46:26.041406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.041377    6777 scope.go:122] "RemoveContainer" containerID="ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3"
Oct 30 19:46:26.043403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.043381    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-lx44v"]
Oct 30 19:46:26.047323 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.047295    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"]
Oct 30 19:46:26.051012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.050981    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp"]
Oct 30 19:46:26.054150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.054128    6777 scope.go:122] "RemoveContainer" containerID="1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd"
Oct 30 19:46:26.054511 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:26.054485    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd\": container with ID starting with 1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd not found: ID does not exist" containerID="1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd"
Oct 30 19:46:26.054562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.054520    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd"} err="failed to get container status \"1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd\": rpc error: code = NotFound desc = could not find container \"1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd\": container with ID starting with 1fbaac703b554704fb92120ccbb474ae141add32f2bbbb899d4c046bd27380cd not found: ID does not exist"
Oct 30 19:46:26.054562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.054539    6777 scope.go:122] "RemoveContainer" containerID="ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3"
Oct 30 19:46:26.054820 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:26.054799    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3\": container with ID starting with ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3 not found: ID does not exist" containerID="ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3"
Oct 30 19:46:26.054888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.054831    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3"} err="failed to get container status \"ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3\": rpc error: code = NotFound desc = could not find container \"ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3\": container with ID starting with ca886e10302e907f49873a8eb4e81deb66fa280c589981820edccabe3f2597e3 not found: ID does not exist"
Oct 30 19:46:26.054888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.054856    6777 scope.go:122] "RemoveContainer" containerID="4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c"
Oct 30 19:46:26.066988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.066960    6777 scope.go:122] "RemoveContainer" containerID="4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c"
Oct 30 19:46:26.067416 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:26.067390    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c\": container with ID starting with 4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c not found: ID does not exist" containerID="4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c"
Oct 30 19:46:26.067490 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.067427    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c"} err="failed to get container status \"4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c\": rpc error: code = NotFound desc = could not find container \"4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c\": container with ID starting with 4338a77a503a199817f20b058f0b8ad6f5560f091ef722e0cdee3344ca6bb37c not found: ID does not exist"
Oct 30 19:46:26.114766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.114731    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" path="/var/lib/kubelet/pods/3c9e3c06-7423-46e4-b910-bba1a9147b70/volumes"
Oct 30 19:46:26.115515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.115493    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" path="/var/lib/kubelet/pods/7e94e090-5bce-47cf-9c75-a776e08b3265/volumes"
Oct 30 19:46:26.116230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.116213    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" path="/var/lib/kubelet/pods/adb479ae-22d3-45b4-99d4-d9377e881735/volumes"
Oct 30 19:46:26.301808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:26.301765    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/premium-w1-cfa4bb-kserve-8f77fb569-zlzbp" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:46:27.536245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.536212    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"]
Oct 30 19:46:27.537000 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.536986    6777 cpu_manager.go:414] "containerMap: removing container" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy"
Oct 30 19:46:27.537087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537001    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy"
Oct 30 19:46:27.537087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537017    6777 cpu_manager.go:414] "containerMap: removing container" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy"
Oct 30 19:46:27.537087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537023    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09659fd9-9629-41e5-9e07-e6680dfa05e4" containerName="istio-proxy"
Oct 30 19:46:27.537087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537037    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1518a909-687d-4d36-bdc7-58a016b1e661" containerName="payload-processing"
Oct 30 19:46:27.537087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537043    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1518a909-687d-4d36-bdc7-58a016b1e661" containerName="payload-processing"
Oct 30 19:46:27.537087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537074    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy"
Oct 30 19:46:27.537087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537080    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy"
Oct 30 19:46:27.537304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537116    6777 cpu_manager.go:414] "containerMap: removing container" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main"
Oct 30 19:46:27.537304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537122    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main"
Oct 30 19:46:27.537304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537229    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9e3c06-7423-46e4-b910-bba1a9147b70" containerName="main"
Oct 30 19:46:27.537304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537239    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy"
Oct 30 19:46:27.537304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537247    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1518a909-687d-4d36-bdc7-58a016b1e661" containerName="payload-processing"
Oct 30 19:46:27.537304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537255    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e94e090-5bce-47cf-9c75-a776e08b3265" containerName="istio-proxy"
Oct 30 19:46:27.537304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.537263    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy"
Oct 30 19:46:27.542872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.542849    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.550568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.550537    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"]
Oct 30 19:46:27.681081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.675881    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.681081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.675959    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.681081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.676034    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfnvv\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-kube-api-access-bfnvv\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.681081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.676097    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86401767-2d4b-430d-bfd1-11208842daca-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.681081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.676316    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.681081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.676342    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.681081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.676412    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86401767-2d4b-430d-bfd1-11208842daca-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.681081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.676514    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.681081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.676640    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.864198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.798695    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfnvv\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-kube-api-access-bfnvv\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.864198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.798758    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86401767-2d4b-430d-bfd1-11208842daca-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.864198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.798925    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.864198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.798951    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.864198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.799027    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86401767-2d4b-430d-bfd1-11208842daca-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.864198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.799143    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.864198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.841872    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.864198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.863780    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.876564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.865745    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86401767-2d4b-430d-bfd1-11208842daca-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.878952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.877584    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.878952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.877688    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.878952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.877737    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.878952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.878387    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.878952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.878609    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.880925 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.880862    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.893680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.892174    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86401767-2d4b-430d-bfd1-11208842daca-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.894991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.894927    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfnvv\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-kube-api-access-bfnvv\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:27.923285 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:27.923250    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:28.174018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:28.168171    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:28.589832 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:46:28.589495    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86401767_2d4b_430d_bfd1_11208842daca.slice/crio-5f7e98e112f6b57c984a2e4e4c3002a2c0c8b5816eeddfa810dbabaca0efc10f WatchSource:0}: Error finding container 5f7e98e112f6b57c984a2e4e4c3002a2c0c8b5816eeddfa810dbabaca0efc10f: Status 404 returned error can't find the container with id 5f7e98e112f6b57c984a2e4e4c3002a2c0c8b5816eeddfa810dbabaca0efc10f
Oct 30 19:46:28.598075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:28.597464    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:28.598075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:28.597624    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:28.598075 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:28.597733    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:28.613421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:28.612869    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"]
Oct 30 19:46:29.084702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:29.082253    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" event={"ID":"86401767-2d4b-430d-bfd1-11208842daca","Type":"ContainerStarted","Data":"0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b"}
Oct 30 19:46:29.084702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:29.082334    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" event={"ID":"86401767-2d4b-430d-bfd1-11208842daca","Type":"ContainerStarted","Data":"5f7e98e112f6b57c984a2e4e4c3002a2c0c8b5816eeddfa810dbabaca0efc10f"}
Oct 30 19:46:29.133025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:29.132953    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podStartSLOduration=2.132933777 podStartE2EDuration="2.132933777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:29.12857097 +0000 UTC m=+2371.125486353" watchObservedRunningTime="2026-10-30 19:46:29.132933777 +0000 UTC m=+2371.129849165"
Oct 30 19:46:29.169627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:29.169587    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:29.171872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:29.171793    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 19:46:29.171872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:29.171854    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 19:46:30.176305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:30.176248    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 19:46:30.176848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:30.176337    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 19:46:30.402962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:30.402927    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"]
Oct 30 19:46:31.187430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:31.186868    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 19:46:31.187430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:31.186925    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 19:46:32.178079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:32.169465    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 19:46:32.178079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:32.169526    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 19:46:33.174086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:33.173554    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 19:46:33.174086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:33.173625    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 19:46:34.187108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:34.184693    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 19:46:34.197096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:34.193130    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 19:46:35.170529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:35.170488    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 19:46:35.171506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:35.171467    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 19:46:35.404221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:35.404141    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz" podUID="8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" containerName="payload-pre-processing" containerID="cri-o://6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57" gracePeriod=25
Oct 30 19:46:35.608022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:35.607978    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"]
Oct 30 19:46:35.633896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:35.633847    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"]
Oct 30 19:46:36.179138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:36.179100    6777 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 19:46:36.179999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:36.179494    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 19:46:37.180581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:37.180542    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:37.182845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:37.182815    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:37.190782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:37.190745    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:46:40.609111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:40.609037    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx" podUID="88579cc7-45cc-441d-bd4e-15cf9cacc928" containerName="payload-processing" containerID="cri-o://05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153" gracePeriod=25
Oct 30 19:46:40.690704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:40.680535    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th" podUID="82f21a64-764f-45fc-bc16-bc7619cf1694" containerName="payload-pre-processing" containerID="cri-o://0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92" gracePeriod=25
Oct 30 19:46:41.041187 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.041160    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:46:41.045210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.045183    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:46:41.100208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.100165    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f21a64-764f-45fc-bc16-bc7619cf1694-kube-api-access-bflh5\" (UniqueName: \"kubernetes.io/projected/82f21a64-764f-45fc-bc16-bc7619cf1694-kube-api-access-bflh5\") pod \"82f21a64-764f-45fc-bc16-bc7619cf1694\" (UID: \"82f21a64-764f-45fc-bc16-bc7619cf1694\") "
Oct 30 19:46:41.100415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.100302    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f21a64-764f-45fc-bc16-bc7619cf1694-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f21a64-764f-45fc-bc16-bc7619cf1694-plugins-config-volume\") pod \"82f21a64-764f-45fc-bc16-bc7619cf1694\" (UID: \"82f21a64-764f-45fc-bc16-bc7619cf1694\") "
Oct 30 19:46:41.100415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.100347    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88579cc7-45cc-441d-bd4e-15cf9cacc928-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88579cc7-45cc-441d-bd4e-15cf9cacc928-plugins-config-volume\") pod \"88579cc7-45cc-441d-bd4e-15cf9cacc928\" (UID: \"88579cc7-45cc-441d-bd4e-15cf9cacc928\") "
Oct 30 19:46:41.100549 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.100428    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88579cc7-45cc-441d-bd4e-15cf9cacc928-kube-api-access-fsddt\" (UniqueName: \"kubernetes.io/projected/88579cc7-45cc-441d-bd4e-15cf9cacc928-kube-api-access-fsddt\") pod \"88579cc7-45cc-441d-bd4e-15cf9cacc928\" (UID: \"88579cc7-45cc-441d-bd4e-15cf9cacc928\") "
Oct 30 19:46:41.100734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.100704    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f21a64-764f-45fc-bc16-bc7619cf1694-plugins-config-volume" pod "82f21a64-764f-45fc-bc16-bc7619cf1694" (UID: "82f21a64-764f-45fc-bc16-bc7619cf1694"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:41.100804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.100740    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88579cc7-45cc-441d-bd4e-15cf9cacc928-plugins-config-volume" pod "88579cc7-45cc-441d-bd4e-15cf9cacc928" (UID: "88579cc7-45cc-441d-bd4e-15cf9cacc928"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:41.101246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.101223    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f21a64-764f-45fc-bc16-bc7619cf1694-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:41.101246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.101243    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88579cc7-45cc-441d-bd4e-15cf9cacc928-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:41.102486 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.102461    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f21a64-764f-45fc-bc16-bc7619cf1694-kube-api-access-bflh5" pod "82f21a64-764f-45fc-bc16-bc7619cf1694" (UID: "82f21a64-764f-45fc-bc16-bc7619cf1694"). InnerVolumeSpecName "kube-api-access-bflh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:41.102760 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.102740    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88579cc7-45cc-441d-bd4e-15cf9cacc928-kube-api-access-fsddt" pod "88579cc7-45cc-441d-bd4e-15cf9cacc928" (UID: "88579cc7-45cc-441d-bd4e-15cf9cacc928"). InnerVolumeSpecName "kube-api-access-fsddt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:41.202971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.202885    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bflh5\" (UniqueName: \"kubernetes.io/projected/82f21a64-764f-45fc-bc16-bc7619cf1694-kube-api-access-bflh5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:41.202971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.202926    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsddt\" (UniqueName: \"kubernetes.io/projected/88579cc7-45cc-441d-bd4e-15cf9cacc928-kube-api-access-fsddt\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:41.356863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.356820    6777 generic.go:356] "Generic (PLEG): container finished" podID="88579cc7-45cc-441d-bd4e-15cf9cacc928" containerID="05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153" exitCode=0
Oct 30 19:46:41.357352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.357328    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"
Oct 30 19:46:41.357586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.357320    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx" event={"ID":"88579cc7-45cc-441d-bd4e-15cf9cacc928","Type":"ContainerDied","Data":"05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153"}
Oct 30 19:46:41.357869 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.357844    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx" event={"ID":"88579cc7-45cc-441d-bd4e-15cf9cacc928","Type":"ContainerDied","Data":"6f78bf6b51d2ae232a0966d76e7d718015761ef24b5958fcfe370c93813032b3"}
Oct 30 19:46:41.358051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.357917    6777 scope.go:122] "RemoveContainer" containerID="05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153"
Oct 30 19:46:41.359663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.359593    6777 generic.go:356] "Generic (PLEG): container finished" podID="82f21a64-764f-45fc-bc16-bc7619cf1694" containerID="0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92" exitCode=0
Oct 30 19:46:41.359901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.359878    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th" event={"ID":"82f21a64-764f-45fc-bc16-bc7619cf1694","Type":"ContainerDied","Data":"0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92"}
Oct 30 19:46:41.359988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.359923    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"
Oct 30 19:46:41.359988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.359945    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th" event={"ID":"82f21a64-764f-45fc-bc16-bc7619cf1694","Type":"ContainerDied","Data":"644b9d95ac8fe4c62bc05b10bdd720cf0d976c36026fc6ee62482d827330186d"}
Oct 30 19:46:41.372893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.372863    6777 scope.go:122] "RemoveContainer" containerID="05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153"
Oct 30 19:46:41.373328 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:41.373301    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153\": container with ID starting with 05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153 not found: ID does not exist" containerID="05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153"
Oct 30 19:46:41.373424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.373336    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153"} err="failed to get container status \"05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153\": rpc error: code = NotFound desc = could not find container \"05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153\": container with ID starting with 05d119ed4915ee434b8a60156ed6145931a8d0b7a3ecf420a026aafbb629e153 not found: ID does not exist"
Oct 30 19:46:41.373424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.373353    6777 scope.go:122] "RemoveContainer" containerID="0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92"
Oct 30 19:46:41.386886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.386858    6777 scope.go:122] "RemoveContainer" containerID="0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92"
Oct 30 19:46:41.387323 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:41.387301    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92\": container with ID starting with 0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92 not found: ID does not exist" containerID="0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92"
Oct 30 19:46:41.387427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.387332    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92"} err="failed to get container status \"0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92\": rpc error: code = NotFound desc = could not find container \"0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92\": container with ID starting with 0d5d2a750a81d9ab4ebf61b57488b488b971d9cddd62aebdaab7398f0611fb92 not found: ID does not exist"
Oct 30 19:46:41.396723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.396516    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"]
Oct 30 19:46:41.401312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.401279    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-76f9d799c9-nt6th"]
Oct 30 19:46:41.417528 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.417488    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"]
Oct 30 19:46:41.421433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:41.421398    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-6b8976b7cb-htnjx"]
Oct 30 19:46:42.115257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:42.115225    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f21a64-764f-45fc-bc16-bc7619cf1694" path="/var/lib/kubelet/pods/82f21a64-764f-45fc-bc16-bc7619cf1694/volumes"
Oct 30 19:46:42.115800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:42.115786    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88579cc7-45cc-441d-bd4e-15cf9cacc928" path="/var/lib/kubelet/pods/88579cc7-45cc-441d-bd4e-15cf9cacc928/volumes"
Oct 30 19:46:43.174472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.174434    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"]
Oct 30 19:46:43.175463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.175435    6777 cpu_manager.go:414] "containerMap: removing container" podUID="88579cc7-45cc-441d-bd4e-15cf9cacc928" containerName="payload-processing"
Oct 30 19:46:43.175463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.175461    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88579cc7-45cc-441d-bd4e-15cf9cacc928" containerName="payload-processing"
Oct 30 19:46:43.175728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.175479    6777 cpu_manager.go:414] "containerMap: removing container" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy"
Oct 30 19:46:43.175728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.175488    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb479ae-22d3-45b4-99d4-d9377e881735" containerName="istio-proxy"
Oct 30 19:46:43.175728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.175506    6777 cpu_manager.go:414] "containerMap: removing container" podUID="82f21a64-764f-45fc-bc16-bc7619cf1694" containerName="payload-pre-processing"
Oct 30 19:46:43.175728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.175513    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f21a64-764f-45fc-bc16-bc7619cf1694" containerName="payload-pre-processing"
Oct 30 19:46:43.175728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.175712    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="88579cc7-45cc-441d-bd4e-15cf9cacc928" containerName="payload-processing"
Oct 30 19:46:43.175728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.175731    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f21a64-764f-45fc-bc16-bc7619cf1694" containerName="payload-pre-processing"
Oct 30 19:46:43.181138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.181115    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.197294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.197204    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"]
Oct 30 19:46:43.244566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.244532    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.244775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.244677    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb75217-842f-45f8-8e88-61453b0f64ed-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.244877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.244787    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.244877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.244832    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z4dq\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-kube-api-access-5z4dq\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.244991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.244974    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.245241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.245218    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.245342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.245274    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.245400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.245336    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.245400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.245387    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.347405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347362    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.347629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347424    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.347629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347442    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.347629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347464    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.347819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347661    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.347819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347796    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb75217-842f-45f8-8e88-61453b0f64ed-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.347922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347834    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.347922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347846    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.347922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347877    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.348079 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.347967    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z4dq\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-kube-api-access-5z4dq\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.348150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.348103    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.348150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.348112    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.348150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.348125    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.348576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.348553    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb75217-842f-45f8-8e88-61453b0f64ed-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.350435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.350411    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.350572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.350556    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.360137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.360109    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z4dq\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-kube-api-access-5z4dq\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.360310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.360154    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.518262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.518168    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:43.658548 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:43.658510    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:46:43.809452 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:46:43.809417    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb75217_842f_45f8_8e88_61453b0f64ed.slice/crio-b77999406f6402fe6284873c964f24b45d0878e98f5827f4c89b21b52d86d421 WatchSource:0}: Error finding container b77999406f6402fe6284873c964f24b45d0878e98f5827f4c89b21b52d86d421: Status 404 returned error can't find the container with id b77999406f6402fe6284873c964f24b45d0878e98f5827f4c89b21b52d86d421
Oct 30 19:46:43.809700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.809681    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"]
Oct 30 19:46:43.812534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.812502    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:43.812623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.812571    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:43.812623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:43.812619    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:46:44.414454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:44.414418    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" event={"ID":"1eb75217-842f-45f8-8e88-61453b0f64ed","Type":"ContainerStarted","Data":"de76158be39008f19c9d4a4a7f319624301b521c2f9dfeb43fb9724ddd567390"}
Oct 30 19:46:44.414966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:44.414476    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" event={"ID":"1eb75217-842f-45f8-8e88-61453b0f64ed","Type":"ContainerStarted","Data":"b77999406f6402fe6284873c964f24b45d0878e98f5827f4c89b21b52d86d421"}
Oct 30 19:46:44.440606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:44.440546    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" podStartSLOduration=1.440525521 podStartE2EDuration="1.440525521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:44.439923853 +0000 UTC m=+2386.436839240" watchObservedRunningTime="2026-10-30 19:46:44.440525521 +0000 UTC m=+2386.437440904"
Oct 30 19:46:44.519047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:44.519005    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:44.520929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:44.520905    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:46:44.521040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:44.520960    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 19:46:44.849443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:44.849401    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz" podUID="8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:46:45.518685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:45.518645    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:46:45.519189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:45.518705    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 19:46:46.530217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:46.521763    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:46:46.530217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:46.521837    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 19:46:47.245955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:47.245919    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"]
Oct 30 19:46:47.246234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:47.246177    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy" containerID="cri-o://adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386" gracePeriod=30
Oct 30 19:46:47.271198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:47.271156    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"]
Oct 30 19:46:47.271715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:47.271681    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy" containerID="cri-o://b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169" gracePeriod=30
Oct 30 19:46:47.285030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:47.284993    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"]
Oct 30 19:46:47.285465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:47.285427    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" containerID="cri-o://9d580967d65f3d21e2627daff29595f31239b50b3b6058b2233164ad617aed72" gracePeriod=30
Oct 30 19:46:47.518929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:47.518838    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:46:47.518929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:47.518894    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 19:46:48.519325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:48.519287    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:46:48.519747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:48.519344    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 19:46:49.519355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:49.519317    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:46:49.519860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:49.519381    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 19:46:50.177170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.177038    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"]
Oct 30 19:46:50.185155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.185122    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"]
Oct 30 19:46:50.185326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.185211    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:46:50.194757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.194509    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa49c5\""
Oct 30 19:46:50.194862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.194848    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aa49c5-dockercfg-8msl4\""
Oct 30 19:46:50.228916 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.228882    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"]
Oct 30 19:46:50.237498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.237474    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"]
Oct 30 19:46:50.237783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.237738    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:46:50.316849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.316812    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdffbd8e-5209-4b25-b923-92e782647e4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd\" (UID: \"cdffbd8e-5209-4b25-b923-92e782647e4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:46:50.317180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.317155    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqcm\" (UniqueName: \"kubernetes.io/projected/cdffbd8e-5209-4b25-b923-92e782647e4b-kube-api-access-fgqcm\") pod \"payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd\" (UID: \"cdffbd8e-5209-4b25-b923-92e782647e4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:46:50.520140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.519902    6777 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:46:50.523718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.523676    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 19:46:50.614912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.614855    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rzn\" (UniqueName: \"kubernetes.io/projected/4a4dd36a-3268-45c8-a86e-a37c70e77081-kube-api-access-l4rzn\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79\" (UID: \"4a4dd36a-3268-45c8-a86e-a37c70e77081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:46:50.617792 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.617657    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:50.617961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.617829    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:50.619195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.619166    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4dd36a-3268-45c8-a86e-a37c70e77081-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79\" (UID: \"4a4dd36a-3268-45c8-a86e-a37c70e77081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:46:50.619363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.619343    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdffbd8e-5209-4b25-b923-92e782647e4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd\" (UID: \"cdffbd8e-5209-4b25-b923-92e782647e4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:46:50.619560 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.619538    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqcm\" (UniqueName: \"kubernetes.io/projected/cdffbd8e-5209-4b25-b923-92e782647e4b-kube-api-access-fgqcm\") pod \"payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd\" (UID: \"cdffbd8e-5209-4b25-b923-92e782647e4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:46:50.621138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.621107    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdffbd8e-5209-4b25-b923-92e782647e4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd\" (UID: \"cdffbd8e-5209-4b25-b923-92e782647e4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:46:50.633424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.633402    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqcm\" (UniqueName: \"kubernetes.io/projected/cdffbd8e-5209-4b25-b923-92e782647e4b-kube-api-access-fgqcm\") pod \"payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd\" (UID: \"cdffbd8e-5209-4b25-b923-92e782647e4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:46:50.721761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.721719    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4dd36a-3268-45c8-a86e-a37c70e77081-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79\" (UID: \"4a4dd36a-3268-45c8-a86e-a37c70e77081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:46:50.721943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.721840    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rzn\" (UniqueName: \"kubernetes.io/projected/4a4dd36a-3268-45c8-a86e-a37c70e77081-kube-api-access-l4rzn\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79\" (UID: \"4a4dd36a-3268-45c8-a86e-a37c70e77081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:46:50.722522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.722500    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4dd36a-3268-45c8-a86e-a37c70e77081-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79\" (UID: \"4a4dd36a-3268-45c8-a86e-a37c70e77081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:46:50.735128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.735100    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rzn\" (UniqueName: \"kubernetes.io/projected/4a4dd36a-3268-45c8-a86e-a37c70e77081-kube-api-access-l4rzn\") pod \"payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79\" (UID: \"4a4dd36a-3268-45c8-a86e-a37c70e77081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:46:50.801362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.801328    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:46:50.980306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:50.977215    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:46:51.112801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.112766    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"]
Oct 30 19:46:51.116735 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:46:51.116701    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdffbd8e_5209_4b25_b923_92e782647e4b.slice/crio-11780c8385eede051843dcc3bdf4cd6e52b1fff54c0a44f94be355f01aa999eb WatchSource:0}: Error finding container 11780c8385eede051843dcc3bdf4cd6e52b1fff54c0a44f94be355f01aa999eb: Status 404 returned error can't find the container with id 11780c8385eede051843dcc3bdf4cd6e52b1fff54c0a44f94be355f01aa999eb
Oct 30 19:46:51.276030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.275999    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"]
Oct 30 19:46:51.279387 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:46:51.279344    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4dd36a_3268_45c8_a86e_a37c70e77081.slice/crio-45ce7f7f63ead5032d6d26dbaf6e3db05e70eb254042fd854ca346e62f2af70b WatchSource:0}: Error finding container 45ce7f7f63ead5032d6d26dbaf6e3db05e70eb254042fd854ca346e62f2af70b: Status 404 returned error can't find the container with id 45ce7f7f63ead5032d6d26dbaf6e3db05e70eb254042fd854ca346e62f2af70b
Oct 30 19:46:51.469349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.469314    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"]
Oct 30 19:46:51.477951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.477901    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"]
Oct 30 19:46:51.528898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.528862    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:51.529538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.529518    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:51.530855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.530831    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:46:51.646318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.646223    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd" event={"ID":"cdffbd8e-5209-4b25-b923-92e782647e4b","Type":"ContainerStarted","Data":"7d3f0613118d3827868551e20f6000ba2b73d82a9e483b7bd884231bd3afe015"}
Oct 30 19:46:51.646318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.646295    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd" event={"ID":"cdffbd8e-5209-4b25-b923-92e782647e4b","Type":"ContainerStarted","Data":"11780c8385eede051843dcc3bdf4cd6e52b1fff54c0a44f94be355f01aa999eb"}
Oct 30 19:46:51.646551 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.646345    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:46:51.650650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.650610    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79" event={"ID":"4a4dd36a-3268-45c8-a86e-a37c70e77081","Type":"ContainerStarted","Data":"6f965a753b6c97692bb4cf10a5eb3b803edda85d281bc3abb103e9df3aa642dd"}
Oct 30 19:46:51.650814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.650695    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79" event={"ID":"4a4dd36a-3268-45c8-a86e-a37c70e77081","Type":"ContainerStarted","Data":"45ce7f7f63ead5032d6d26dbaf6e3db05e70eb254042fd854ca346e62f2af70b"}
Oct 30 19:46:51.650814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.650729    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:46:51.672199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.672145    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd" podStartSLOduration=1.672127892 podStartE2EDuration="1.672127892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:51.670192009 +0000 UTC m=+2393.667107412" watchObservedRunningTime="2026-10-30 19:46:51.672127892 +0000 UTC m=+2393.669043278"
Oct 30 19:46:51.694896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:51.694840    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79" podStartSLOduration=1.6948242439999999 podStartE2EDuration="1.694824244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:51.690861851 +0000 UTC m=+2393.687777235" watchObservedRunningTime="2026-10-30 19:46:51.694824244 +0000 UTC m=+2393.691739627"
Oct 30 19:46:52.518599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.518569    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:46:52.602259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602009    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-envoy\") pod \"b316199b-90ea-49c2-92f9-31a270b16d4a\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") "
Oct 30 19:46:52.602259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602100    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-credential-socket\") pod \"b316199b-90ea-49c2-92f9-31a270b16d4a\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") "
Oct 30 19:46:52.602259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602161    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-certs\") pod \"b316199b-90ea-49c2-92f9-31a270b16d4a\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") "
Oct 30 19:46:52.602773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602310    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-socket\") pod \"b316199b-90ea-49c2-92f9-31a270b16d4a\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") "
Oct 30 19:46:52.602773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602350    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-data\") pod \"b316199b-90ea-49c2-92f9-31a270b16d4a\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") "
Oct 30 19:46:52.602773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602475    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-token\") pod \"b316199b-90ea-49c2-92f9-31a270b16d4a\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") "
Oct 30 19:46:52.602773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602704    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b316199b-90ea-49c2-92f9-31a270b16d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b316199b-90ea-49c2-92f9-31a270b16d4a-istiod-ca-cert\") pod \"b316199b-90ea-49c2-92f9-31a270b16d4a\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") "
Oct 30 19:46:52.602773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602746    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-kube-api-access-q5g7l\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-kube-api-access-q5g7l\") pod \"b316199b-90ea-49c2-92f9-31a270b16d4a\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") "
Oct 30 19:46:52.603033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602774    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-podinfo\") pod \"b316199b-90ea-49c2-92f9-31a270b16d4a\" (UID: \"b316199b-90ea-49c2-92f9-31a270b16d4a\") "
Oct 30 19:46:52.603033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.602876    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-credential-socket" pod "b316199b-90ea-49c2-92f9-31a270b16d4a" (UID: "b316199b-90ea-49c2-92f9-31a270b16d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.603251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.603118    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-data" pod "b316199b-90ea-49c2-92f9-31a270b16d4a" (UID: "b316199b-90ea-49c2-92f9-31a270b16d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.603251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.603123    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-socket" pod "b316199b-90ea-49c2-92f9-31a270b16d4a" (UID: "b316199b-90ea-49c2-92f9-31a270b16d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.605657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.603697    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b316199b-90ea-49c2-92f9-31a270b16d4a-istiod-ca-cert" pod "b316199b-90ea-49c2-92f9-31a270b16d4a" (UID: "b316199b-90ea-49c2-92f9-31a270b16d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:52.605657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.603823    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.605657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.603848    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.605657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.603862    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.605657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.604387    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-certs" pod "b316199b-90ea-49c2-92f9-31a270b16d4a" (UID: "b316199b-90ea-49c2-92f9-31a270b16d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.606727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.606702    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-kube-api-access-q5g7l" pod "b316199b-90ea-49c2-92f9-31a270b16d4a" (UID: "b316199b-90ea-49c2-92f9-31a270b16d4a"). InnerVolumeSpecName "kube-api-access-q5g7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.608326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.608292    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-podinfo" pod "b316199b-90ea-49c2-92f9-31a270b16d4a" (UID: "b316199b-90ea-49c2-92f9-31a270b16d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:52.608588 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.608564    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-token" pod "b316199b-90ea-49c2-92f9-31a270b16d4a" (UID: "b316199b-90ea-49c2-92f9-31a270b16d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.609250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.609228    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-envoy" pod "b316199b-90ea-49c2-92f9-31a270b16d4a" (UID: "b316199b-90ea-49c2-92f9-31a270b16d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.626464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.626433    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:46:52.668725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.665492    6777 generic.go:356] "Generic (PLEG): container finished" podID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerID="adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386" exitCode=0
Oct 30 19:46:52.668725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.665611    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" event={"ID":"b316199b-90ea-49c2-92f9-31a270b16d4a","Type":"ContainerDied","Data":"adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386"}
Oct 30 19:46:52.668725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.665663    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8" event={"ID":"b316199b-90ea-49c2-92f9-31a270b16d4a","Type":"ContainerDied","Data":"9f8f2d10972336214ebdcbcb778757807154ab6dd53eacd3cc12d7ae5706a0e4"}
Oct 30 19:46:52.668725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.665694    6777 scope.go:122] "RemoveContainer" containerID="adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386"
Oct 30 19:46:52.668725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.665874    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"
Oct 30 19:46:52.681702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.681285    6777 generic.go:356] "Generic (PLEG): container finished" podID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerID="9d580967d65f3d21e2627daff29595f31239b50b3b6058b2233164ad617aed72" exitCode=0
Oct 30 19:46:52.681702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.681583    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" event={"ID":"dbed3e02-2c98-4da9-9457-e3fdb48be2de","Type":"ContainerDied","Data":"9d580967d65f3d21e2627daff29595f31239b50b3b6058b2233164ad617aed72"}
Oct 30 19:46:52.688034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.684395    6777 generic.go:356] "Generic (PLEG): container finished" podID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerID="b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169" exitCode=0
Oct 30 19:46:52.688034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.685885    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"
Oct 30 19:46:52.688034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.686045    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" event={"ID":"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a","Type":"ContainerDied","Data":"b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169"}
Oct 30 19:46:52.688034 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.686562    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7" event={"ID":"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a","Type":"ContainerDied","Data":"cd21eca7938cf6d705531f86ca962c51e21d5d22c27517db02b0dd48135f7d3f"}
Oct 30 19:46:52.709497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.707659    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-certs\") pod \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") "
Oct 30 19:46:52.709497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.707778    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-envoy\") pod \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") "
Oct 30 19:46:52.709497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.707903    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-data\") pod \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") "
Oct 30 19:46:52.709497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.707942    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-kube-api-access-nlk68\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-kube-api-access-nlk68\") pod \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") "
Oct 30 19:46:52.709497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.707996    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-token\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-token\") pod \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") "
Oct 30 19:46:52.709497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.708146    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-socket\") pod \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") "
Oct 30 19:46:52.709497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.708203    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istiod-ca-cert\") pod \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") "
Oct 30 19:46:52.709497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.708322    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-credential-socket\") pod \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") "
Oct 30 19:46:52.709497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.708360    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-podinfo\") pod \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\" (UID: \"6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a\") "
Oct 30 19:46:52.710080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.709767    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b316199b-90ea-49c2-92f9-31a270b16d4a-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.710080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.709802    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5g7l\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-kube-api-access-q5g7l\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.710080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.709818    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.710080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.709832    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.710080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.709845    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b316199b-90ea-49c2-92f9-31a270b16d4a-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.710080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.709861    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b316199b-90ea-49c2-92f9-31a270b16d4a-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.710366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.710323    6777 scope.go:122] "RemoveContainer" containerID="f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb"
Oct 30 19:46:52.711447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.711105    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-socket" pod "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" (UID: "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.712699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.712049    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istiod-ca-cert" pod "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" (UID: "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:52.712699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.712318    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-credential-socket" pod "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" (UID: "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.712699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.712504    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-data" pod "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" (UID: "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.712699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.712669    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-certs" pod "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" (UID: "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.724669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.724618    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-kube-api-access-nlk68" pod "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" (UID: "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a"). InnerVolumeSpecName "kube-api-access-nlk68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.724849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.724694    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"]
Oct 30 19:46:52.732139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.726736    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-envoy" pod "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" (UID: "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.738357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.738288    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-podinfo" pod "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" (UID: "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:52.738702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.738675    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-token" pod "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" (UID: "6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.740685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.740654    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-tx8s8"]
Oct 30 19:46:52.768788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.768762    6777 scope.go:122] "RemoveContainer" containerID="adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386"
Oct 30 19:46:52.770010 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:52.769975    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386\": container with ID starting with adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386 not found: ID does not exist" containerID="adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386"
Oct 30 19:46:52.770188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.770019    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386"} err="failed to get container status \"adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386\": rpc error: code = NotFound desc = could not find container \"adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386\": container with ID starting with adac08dd54beb286fd4f98d4867ec1e00daed9b8794b71ee0ce322f1b4950386 not found: ID does not exist"
Oct 30 19:46:52.770188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.770045    6777 scope.go:122] "RemoveContainer" containerID="f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb"
Oct 30 19:46:52.770437 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:52.770418    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb\": container with ID starting with f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb not found: ID does not exist" containerID="f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb"
Oct 30 19:46:52.770482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.770443    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb"} err="failed to get container status \"f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb\": rpc error: code = NotFound desc = could not find container \"f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb\": container with ID starting with f7a75d5cc3b03d406bded44606cc12ad4df0939c1502f572f7c2ad8628165ccb not found: ID does not exist"
Oct 30 19:46:52.770482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.770462    6777 scope.go:122] "RemoveContainer" containerID="b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169"
Oct 30 19:46:52.770604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.770583    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:46:52.784456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.784433    6777 scope.go:122] "RemoveContainer" containerID="b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169"
Oct 30 19:46:52.785245 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:52.785216    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169\": container with ID starting with b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169 not found: ID does not exist" containerID="b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169"
Oct 30 19:46:52.785363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.785257    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169"} err="failed to get container status \"b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169\": rpc error: code = NotFound desc = could not find container \"b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169\": container with ID starting with b94154ec927c05e8587dbedf8b8bb3d443a3cd2ef5db633c993b414ad8bc4169 not found: ID does not exist"
Oct 30 19:46:52.841192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.841162    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.841482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.841462    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.841604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.841592    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.841686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.841675    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlk68\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-kube-api-access-nlk68\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.841755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.841746    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.841828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.841818    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.841904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.841894    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.841982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.841972    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.842049 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.842040    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.943426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943337    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-certs\") pod \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") "
Oct 30 19:46:52.943426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943412    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-podinfo\") pod \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") "
Oct 30 19:46:52.943728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943480    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-data\") pod \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") "
Oct 30 19:46:52.943728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943512    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-envoy\") pod \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") "
Oct 30 19:46:52.943728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943562    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-credential-socket\") pod \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") "
Oct 30 19:46:52.943728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943597    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-token\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-token\") pod \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") "
Oct 30 19:46:52.943728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943647    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istiod-ca-cert\") pod \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") "
Oct 30 19:46:52.943728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943664    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-certs" pod "dbed3e02-2c98-4da9-9457-e3fdb48be2de" (UID: "dbed3e02-2c98-4da9-9457-e3fdb48be2de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.944113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943814    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-kube-api-access-gws76\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-kube-api-access-gws76\") pod \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") "
Oct 30 19:46:52.944113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.943969    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-socket\") pod \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\" (UID: \"dbed3e02-2c98-4da9-9457-e3fdb48be2de\") "
Oct 30 19:46:52.944354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.944333    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-data" pod "dbed3e02-2c98-4da9-9457-e3fdb48be2de" (UID: "dbed3e02-2c98-4da9-9457-e3fdb48be2de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.944634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.944582    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-socket" pod "dbed3e02-2c98-4da9-9457-e3fdb48be2de" (UID: "dbed3e02-2c98-4da9-9457-e3fdb48be2de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.944745 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.944655    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-credential-socket" pod "dbed3e02-2c98-4da9-9457-e3fdb48be2de" (UID: "dbed3e02-2c98-4da9-9457-e3fdb48be2de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.945256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.945228    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.945365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.945259    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.945365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.945275    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.945365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.945290    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.945950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.945910    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istiod-ca-cert" pod "dbed3e02-2c98-4da9-9457-e3fdb48be2de" (UID: "dbed3e02-2c98-4da9-9457-e3fdb48be2de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:52.947005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.946978    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-envoy" pod "dbed3e02-2c98-4da9-9457-e3fdb48be2de" (UID: "dbed3e02-2c98-4da9-9457-e3fdb48be2de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.947343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.947316    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-podinfo" pod "dbed3e02-2c98-4da9-9457-e3fdb48be2de" (UID: "dbed3e02-2c98-4da9-9457-e3fdb48be2de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:52.947598 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.947576    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-token" pod "dbed3e02-2c98-4da9-9457-e3fdb48be2de" (UID: "dbed3e02-2c98-4da9-9457-e3fdb48be2de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.947707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:52.947686    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-kube-api-access-gws76" pod "dbed3e02-2c98-4da9-9457-e3fdb48be2de" (UID: "dbed3e02-2c98-4da9-9457-e3fdb48be2de"). InnerVolumeSpecName "kube-api-access-gws76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:53.047813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.047754    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:53.047813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.047794    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:53.047813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.047809    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:53.047813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.047826    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbed3e02-2c98-4da9-9457-e3fdb48be2de-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:53.048329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.047841    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gws76\" (UniqueName: \"kubernetes.io/projected/dbed3e02-2c98-4da9-9457-e3fdb48be2de-kube-api-access-gws76\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:53.077351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.077267    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"]
Oct 30 19:46:53.083115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.083028    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-6dsl7"]
Oct 30 19:46:53.353372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.353338    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:46:53.698623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.698526    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5" event={"ID":"dbed3e02-2c98-4da9-9457-e3fdb48be2de","Type":"ContainerDied","Data":"78861789841cd896d1554fedbdee4e3802ae2ecef5f0dd71c7784593a9060fd7"}
Oct 30 19:46:53.701900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.701875    6777 scope.go:122] "RemoveContainer" containerID="9d580967d65f3d21e2627daff29595f31239b50b3b6058b2233164ad617aed72"
Oct 30 19:46:53.702206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.699619    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"
Oct 30 19:46:53.762355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.762316    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"]
Oct 30 19:46:53.769773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:53.769737    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qtlz5"]
Oct 30 19:46:54.145082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:54.143712    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" path="/var/lib/kubelet/pods/6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a/volumes"
Oct 30 19:46:54.145082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:54.144874    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" path="/var/lib/kubelet/pods/b316199b-90ea-49c2-92f9-31a270b16d4a/volumes"
Oct 30 19:46:54.146353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:54.145887    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" path="/var/lib/kubelet/pods/dbed3e02-2c98-4da9-9457-e3fdb48be2de/volumes"
Oct 30 19:46:54.872693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:54.872158    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz" podUID="8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:46:55.746911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.746883    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:46:55.793577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.793543    6777 generic.go:356] "Generic (PLEG): container finished" podID="8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" containerID="6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57" exitCode=0
Oct 30 19:46:55.793782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.793597    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"
Oct 30 19:46:55.793782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.793596    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz" event={"ID":"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4","Type":"ContainerDied","Data":"6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57"}
Oct 30 19:46:55.793782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.793669    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz" event={"ID":"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4","Type":"ContainerDied","Data":"6c74b985dfdf97bffef15200c1d131d34a8e38c639dbeb982cef14e2dcb74e64"}
Oct 30 19:46:55.793782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.793700    6777 scope.go:122] "RemoveContainer" containerID="6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57"
Oct 30 19:46:55.808380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.808353    6777 scope.go:122] "RemoveContainer" containerID="6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57"
Oct 30 19:46:55.808813 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:55.808758    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57\": container with ID starting with 6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57 not found: ID does not exist" containerID="6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57"
Oct 30 19:46:55.808948 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.808822    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57"} err="failed to get container status \"6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57\": rpc error: code = NotFound desc = could not find container \"6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57\": container with ID starting with 6114a72d31c3c55e771259fe285b68f327e9e98dcdc35646528c0865c6e1ab57 not found: ID does not exist"
Oct 30 19:46:55.880124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.880046    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-kube-api-access-kdpfq\" (UniqueName: \"kubernetes.io/projected/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-kube-api-access-kdpfq\") pod \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\" (UID: \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\") "
Oct 30 19:46:55.880629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.880149    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-plugins-config-volume\") pod \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\" (UID: \"8e4757a5-50a2-4338-a52c-c71f0b5d1bd4\") "
Oct 30 19:46:55.880917 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.880887    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-plugins-config-volume" pod "8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" (UID: "8e4757a5-50a2-4338-a52c-c71f0b5d1bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:55.881221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.881193    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:55.882768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.882743    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-kube-api-access-kdpfq" pod "8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" (UID: "8e4757a5-50a2-4338-a52c-c71f0b5d1bd4"). InnerVolumeSpecName "kube-api-access-kdpfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:55.983288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:55.983204    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdpfq\" (UniqueName: \"kubernetes.io/projected/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4-kube-api-access-kdpfq\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:56.124651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:56.124618    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"]
Oct 30 19:46:56.128351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:56.128329    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e7123-7688d76b9f-zn9kz"]
Oct 30 19:46:56.828652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:56.828612    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"]
Oct 30 19:46:58.116843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:58.116808    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" path="/var/lib/kubelet/pods/8e4757a5-50a2-4338-a52c-c71f0b5d1bd4/volumes"
Oct 30 19:46:59.011557 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.011518    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"]
Oct 30 19:46:59.011801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.011774    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5" podUID="5ed38bc4-b9f6-490b-a2dd-1977e88971bb" containerName="payload-processing" containerID="cri-o://1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e" gracePeriod=30
Oct 30 19:46:59.032726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.032686    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"]
Oct 30 19:46:59.033328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.033036    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z" podUID="3590172e-2fb2-4488-a7b8-bd257dd73d91" containerName="payload-pre-processing" containerID="cri-o://e51819e5597deb4074c790dc499923e443b7f3bd5e9a0b55deccf4d1d4fb4177" gracePeriod=30
Oct 30 19:46:59.622483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.622442    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:46:59.656241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.656015    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-kube-api-access-hh9f2\" (UniqueName: \"kubernetes.io/projected/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-kube-api-access-hh9f2\") pod \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\" (UID: \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\") "
Oct 30 19:46:59.656241 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.656210    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-plugins-config-volume\") pod \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\" (UID: \"5ed38bc4-b9f6-490b-a2dd-1977e88971bb\") "
Oct 30 19:46:59.657161 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.657053    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-plugins-config-volume" pod "5ed38bc4-b9f6-490b-a2dd-1977e88971bb" (UID: "5ed38bc4-b9f6-490b-a2dd-1977e88971bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:59.657294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.657257    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:59.667709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.664033    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-kube-api-access-hh9f2" pod "5ed38bc4-b9f6-490b-a2dd-1977e88971bb" (UID: "5ed38bc4-b9f6-490b-a2dd-1977e88971bb"). InnerVolumeSpecName "kube-api-access-hh9f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:59.774572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.760172    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh9f2\" (UniqueName: \"kubernetes.io/projected/5ed38bc4-b9f6-490b-a2dd-1977e88971bb-kube-api-access-hh9f2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:59.836367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.835824    6777 generic.go:356] "Generic (PLEG): container finished" podID="5ed38bc4-b9f6-490b-a2dd-1977e88971bb" containerID="1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e" exitCode=0
Oct 30 19:46:59.836367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.836212    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5" event={"ID":"5ed38bc4-b9f6-490b-a2dd-1977e88971bb","Type":"ContainerDied","Data":"1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e"}
Oct 30 19:46:59.836367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.836287    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5" event={"ID":"5ed38bc4-b9f6-490b-a2dd-1977e88971bb","Type":"ContainerDied","Data":"0a9e38a4d595e4264f265971bfcae37d4f55067b33355bc260f4de91dd9831f6"}
Oct 30 19:46:59.836367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.836318    6777 scope.go:122] "RemoveContainer" containerID="1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e"
Oct 30 19:46:59.836725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.836463    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"
Oct 30 19:46:59.845282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.845251    6777 generic.go:356] "Generic (PLEG): container finished" podID="3590172e-2fb2-4488-a7b8-bd257dd73d91" containerID="e51819e5597deb4074c790dc499923e443b7f3bd5e9a0b55deccf4d1d4fb4177" exitCode=0
Oct 30 19:46:59.845567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.845477    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z" event={"ID":"3590172e-2fb2-4488-a7b8-bd257dd73d91","Type":"ContainerDied","Data":"e51819e5597deb4074c790dc499923e443b7f3bd5e9a0b55deccf4d1d4fb4177"}
Oct 30 19:46:59.872714 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.872633    6777 scope.go:122] "RemoveContainer" containerID="1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e"
Oct 30 19:46:59.873111 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:46:59.873024    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e\": container with ID starting with 1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e not found: ID does not exist" containerID="1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e"
Oct 30 19:46:59.873111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.873052    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e"} err="failed to get container status \"1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e\": rpc error: code = NotFound desc = could not find container \"1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e\": container with ID starting with 1b500b80e7657d61c55c78de95c4f20c0448bb7648c0542cfd4b38e47911cf1e not found: ID does not exist"
Oct 30 19:46:59.875136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.875116    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:46:59.884968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.884940    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"]
Oct 30 19:46:59.898443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.898408    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a844e-59d5b45d66-rjhs5"]
Oct 30 19:46:59.966389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.966344    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3590172e-2fb2-4488-a7b8-bd257dd73d91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3590172e-2fb2-4488-a7b8-bd257dd73d91-plugins-config-volume\") pod \"3590172e-2fb2-4488-a7b8-bd257dd73d91\" (UID: \"3590172e-2fb2-4488-a7b8-bd257dd73d91\") "
Oct 30 19:46:59.966604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.966409    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3590172e-2fb2-4488-a7b8-bd257dd73d91-kube-api-access-bcfqj\" (UniqueName: \"kubernetes.io/projected/3590172e-2fb2-4488-a7b8-bd257dd73d91-kube-api-access-bcfqj\") pod \"3590172e-2fb2-4488-a7b8-bd257dd73d91\" (UID: \"3590172e-2fb2-4488-a7b8-bd257dd73d91\") "
Oct 30 19:46:59.966891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.966858    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3590172e-2fb2-4488-a7b8-bd257dd73d91-plugins-config-volume" pod "3590172e-2fb2-4488-a7b8-bd257dd73d91" (UID: "3590172e-2fb2-4488-a7b8-bd257dd73d91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:59.967418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.967333    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3590172e-2fb2-4488-a7b8-bd257dd73d91-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:59.969536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:46:59.969498    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3590172e-2fb2-4488-a7b8-bd257dd73d91-kube-api-access-bcfqj" pod "3590172e-2fb2-4488-a7b8-bd257dd73d91" (UID: "3590172e-2fb2-4488-a7b8-bd257dd73d91"). InnerVolumeSpecName "kube-api-access-bcfqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:00.086021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:00.085977    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcfqj\" (UniqueName: \"kubernetes.io/projected/3590172e-2fb2-4488-a7b8-bd257dd73d91-kube-api-access-bcfqj\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:00.118198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:00.118048    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed38bc4-b9f6-490b-a2dd-1977e88971bb" path="/var/lib/kubelet/pods/5ed38bc4-b9f6-490b-a2dd-1977e88971bb/volumes"
Oct 30 19:47:00.856128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:00.856087    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z" event={"ID":"3590172e-2fb2-4488-a7b8-bd257dd73d91","Type":"ContainerDied","Data":"dfbe684fbeda85db32aef3e3739aadaf2540b08abb8b2e77bd2eba57ee2a8843"}
Oct 30 19:47:00.856653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:00.856144    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"
Oct 30 19:47:00.856653 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:00.856160    6777 scope.go:122] "RemoveContainer" containerID="e51819e5597deb4074c790dc499923e443b7f3bd5e9a0b55deccf4d1d4fb4177"
Oct 30 19:47:00.877203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:00.877175    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"]
Oct 30 19:47:00.885023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:00.884993    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-7b9465b795-t442z"]
Oct 30 19:47:01.829535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:01.829484    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896" podUID="1f430439-89da-4b5a-9c8e-4bb5997ad43d" containerName="payload-pre-processing" containerID="cri-o://c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2" gracePeriod=25
Oct 30 19:47:02.114974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:02.114888    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3590172e-2fb2-4488-a7b8-bd257dd73d91" path="/var/lib/kubelet/pods/3590172e-2fb2-4488-a7b8-bd257dd73d91/volumes"
Oct 30 19:47:02.689862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:02.689829    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:47:02.863875 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:02.863831    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"]
Oct 30 19:47:02.864144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:02.864103    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy" containerID="cri-o://7b5e16a3718a23d71b1abff2557a8e02da2355d2075c9216d0b3f95edd55371a" gracePeriod=30
Oct 30 19:47:03.704298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:03.704242    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:47:04.510582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.510547    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896" podUID="1f430439-89da-4b5a-9c8e-4bb5997ad43d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:04.673874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.673837    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"]
Oct 30 19:47:04.674731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674709    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" containerName="payload-pre-processing"
Oct 30 19:47:04.674811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674733    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" containerName="payload-pre-processing"
Oct 30 19:47:04.674811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674769    6777 cpu_manager.go:414] "containerMap: removing container" podUID="5ed38bc4-b9f6-490b-a2dd-1977e88971bb" containerName="payload-processing"
Oct 30 19:47:04.674811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674777    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed38bc4-b9f6-490b-a2dd-1977e88971bb" containerName="payload-processing"
Oct 30 19:47:04.674811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674796    6777 cpu_manager.go:414] "containerMap: removing container" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy"
Oct 30 19:47:04.674811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674805    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy"
Oct 30 19:47:04.675002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674829    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy"
Oct 30 19:47:04.675002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674836    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy"
Oct 30 19:47:04.675002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674852    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy"
Oct 30 19:47:04.675002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674859    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy"
Oct 30 19:47:04.675002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674877    6777 cpu_manager.go:414] "containerMap: removing container" podUID="3590172e-2fb2-4488-a7b8-bd257dd73d91" containerName="payload-pre-processing"
Oct 30 19:47:04.675002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674883    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3590172e-2fb2-4488-a7b8-bd257dd73d91" containerName="payload-pre-processing"
Oct 30 19:47:04.675002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674903    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy"
Oct 30 19:47:04.675002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.674910    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy"
Oct 30 19:47:04.675338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.675106    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aadc4f7-bda3-4d70-9f64-7aab41fe0f7a" containerName="istio-proxy"
Oct 30 19:47:04.675338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.675124    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="3590172e-2fb2-4488-a7b8-bd257dd73d91" containerName="payload-pre-processing"
Oct 30 19:47:04.675338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.675135    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy"
Oct 30 19:47:04.675338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.675152    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4757a5-50a2-4338-a52c-c71f0b5d1bd4" containerName="payload-pre-processing"
Oct 30 19:47:04.675338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.675165    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed38bc4-b9f6-490b-a2dd-1977e88971bb" containerName="payload-processing"
Oct 30 19:47:04.675338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.675177    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbed3e02-2c98-4da9-9457-e3fdb48be2de" containerName="istio-proxy"
Oct 30 19:47:04.681009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.680987    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:04.689236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.689205    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"]
Oct 30 19:47:04.696546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.695012    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c0a27e\""
Oct 30 19:47:04.748780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.748744    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"]
Oct 30 19:47:04.749762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.749743    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b316199b-90ea-49c2-92f9-31a270b16d4a" containerName="istio-proxy"
Oct 30 19:47:04.753753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.753727    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:04.768456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.768410    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c0a27e-dockercfg-g8mvq\""
Oct 30 19:47:04.770672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.770644    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"]
Oct 30 19:47:04.792394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.792357    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9rnr\" (UniqueName: \"kubernetes.io/projected/43f115ae-b735-46cd-a150-2c525e2f325d-kube-api-access-g9rnr\") pod \"payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk\" (UID: \"43f115ae-b735-46cd-a150-2c525e2f325d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:04.792645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.792621    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f115ae-b735-46cd-a150-2c525e2f325d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk\" (UID: \"43f115ae-b735-46cd-a150-2c525e2f325d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:04.899095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.898433    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f115ae-b735-46cd-a150-2c525e2f325d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk\" (UID: \"43f115ae-b735-46cd-a150-2c525e2f325d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:04.899095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.898678    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9rnr\" (UniqueName: \"kubernetes.io/projected/43f115ae-b735-46cd-a150-2c525e2f325d-kube-api-access-g9rnr\") pod \"payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk\" (UID: \"43f115ae-b735-46cd-a150-2c525e2f325d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:04.899095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.898729    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/0b305198-770f-4d2b-8f37-edccc157cddf-kube-api-access-zn7mf\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2\" (UID: \"0b305198-770f-4d2b-8f37-edccc157cddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:04.899095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.898916    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b305198-770f-4d2b-8f37-edccc157cddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2\" (UID: \"0b305198-770f-4d2b-8f37-edccc157cddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:04.983091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:04.920047    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f115ae-b735-46cd-a150-2c525e2f325d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk\" (UID: \"43f115ae-b735-46cd-a150-2c525e2f325d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:05.063419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:05.063369    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/0b305198-770f-4d2b-8f37-edccc157cddf-kube-api-access-zn7mf\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2\" (UID: \"0b305198-770f-4d2b-8f37-edccc157cddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:05.064002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:05.063974    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b305198-770f-4d2b-8f37-edccc157cddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2\" (UID: \"0b305198-770f-4d2b-8f37-edccc157cddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:05.065017 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:05.064996    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b305198-770f-4d2b-8f37-edccc157cddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2\" (UID: \"0b305198-770f-4d2b-8f37-edccc157cddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:05.066406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:05.066385    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9rnr\" (UniqueName: \"kubernetes.io/projected/43f115ae-b735-46cd-a150-2c525e2f325d-kube-api-access-g9rnr\") pod \"payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk\" (UID: \"43f115ae-b735-46cd-a150-2c525e2f325d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:05.078444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:05.078387    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/0b305198-770f-4d2b-8f37-edccc157cddf-kube-api-access-zn7mf\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2\" (UID: \"0b305198-770f-4d2b-8f37-edccc157cddf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:05.301712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:05.301682    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:05.390123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:05.386632    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:05.648857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:05.648758    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"]
Oct 30 19:47:05.919728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:05.919695    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"]
Oct 30 19:47:05.934748 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:47:05.934716    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b305198_770f_4d2b_8f37_edccc157cddf.slice/crio-8275976852d1d16c6669063c2d8007984602d527ef2d0d8b6f976c504a44399b WatchSource:0}: Error finding container 8275976852d1d16c6669063c2d8007984602d527ef2d0d8b6f976c504a44399b: Status 404 returned error can't find the container with id 8275976852d1d16c6669063c2d8007984602d527ef2d0d8b6f976c504a44399b
Oct 30 19:47:06.070207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.070163    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk" event={"ID":"43f115ae-b735-46cd-a150-2c525e2f325d","Type":"ContainerStarted","Data":"18080aa1aebcbd462bd5a3d7fc1997223b328717f3c0072c884bf7a9dfd1a122"}
Oct 30 19:47:06.070518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.070235    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk" event={"ID":"43f115ae-b735-46cd-a150-2c525e2f325d","Type":"ContainerStarted","Data":"eaa88a8d584053a66b108a7e926497af6afbe2f8fe463139bdd89d8de52028a5"}
Oct 30 19:47:06.070518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.070304    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:06.072827 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.072797    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2" event={"ID":"0b305198-770f-4d2b-8f37-edccc157cddf","Type":"ContainerStarted","Data":"3b24869fd31e9a0e2cbc8cd9ae6884b08497350f5fa6979b3175053b52f8ca92"}
Oct 30 19:47:06.072962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.072856    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2" event={"ID":"0b305198-770f-4d2b-8f37-edccc157cddf","Type":"ContainerStarted","Data":"8275976852d1d16c6669063c2d8007984602d527ef2d0d8b6f976c504a44399b"}
Oct 30 19:47:06.072962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.072886    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:06.094279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.094223    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk" podStartSLOduration=2.094207895 podStartE2EDuration="2.094207895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:06.093086255 +0000 UTC m=+2408.090001638" watchObservedRunningTime="2026-10-30 19:47:06.094207895 +0000 UTC m=+2408.091123279"
Oct 30 19:47:06.119377 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.119311    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2" podStartSLOduration=2.119293996 podStartE2EDuration="2.119293996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:06.116424514 +0000 UTC m=+2408.113339898" watchObservedRunningTime="2026-10-30 19:47:06.119293996 +0000 UTC m=+2408.116209382"
Oct 30 19:47:06.506008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.505828    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main" containerID="cri-o://dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651" gracePeriod=45
Oct 30 19:47:06.519494 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.519454    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main" containerID="cri-o://27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6" gracePeriod=45
Oct 30 19:47:06.853909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.853870    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:47:06.889013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.888980    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:47:06.895144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.894726    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b71be04-6fbe-4637-a36b-ff16915906f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b71be04-6fbe-4637-a36b-ff16915906f2-tls-certs\") pod \"3b71be04-6fbe-4637-a36b-ff16915906f2\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") "
Oct 30 19:47:06.895144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.894800    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-tmp-dir\") pod \"3b71be04-6fbe-4637-a36b-ff16915906f2\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") "
Oct 30 19:47:06.895144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.894856    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b71be04-6fbe-4637-a36b-ff16915906f2-kube-api-access-z5z5g\" (UniqueName: \"kubernetes.io/projected/3b71be04-6fbe-4637-a36b-ff16915906f2-kube-api-access-z5z5g\") pod \"3b71be04-6fbe-4637-a36b-ff16915906f2\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") "
Oct 30 19:47:06.895470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.895185    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-dshm\") pod \"3b71be04-6fbe-4637-a36b-ff16915906f2\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") "
Oct 30 19:47:06.895470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.895339    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-home\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-home\") pod \"3b71be04-6fbe-4637-a36b-ff16915906f2\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") "
Oct 30 19:47:06.895470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.895386    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-model-cache\") pod \"3b71be04-6fbe-4637-a36b-ff16915906f2\" (UID: \"3b71be04-6fbe-4637-a36b-ff16915906f2\") "
Oct 30 19:47:06.895633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.895522    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-tmp-dir" pod "3b71be04-6fbe-4637-a36b-ff16915906f2" (UID: "3b71be04-6fbe-4637-a36b-ff16915906f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:06.896518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.895959    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-home" pod "3b71be04-6fbe-4637-a36b-ff16915906f2" (UID: "3b71be04-6fbe-4637-a36b-ff16915906f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:06.896518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.896069    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-model-cache" pod "3b71be04-6fbe-4637-a36b-ff16915906f2" (UID: "3b71be04-6fbe-4637-a36b-ff16915906f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:06.896518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.896456    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:06.896518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.896478    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:06.896518 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.896494    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:06.898668 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.898642    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b71be04-6fbe-4637-a36b-ff16915906f2-tls-certs" pod "3b71be04-6fbe-4637-a36b-ff16915906f2" (UID: "3b71be04-6fbe-4637-a36b-ff16915906f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:06.898814 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.898746    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-dshm" pod "3b71be04-6fbe-4637-a36b-ff16915906f2" (UID: "3b71be04-6fbe-4637-a36b-ff16915906f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:06.907661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.905194    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b71be04-6fbe-4637-a36b-ff16915906f2-kube-api-access-z5z5g" pod "3b71be04-6fbe-4637-a36b-ff16915906f2" (UID: "3b71be04-6fbe-4637-a36b-ff16915906f2"). InnerVolumeSpecName "kube-api-access-z5z5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:06.997338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.997308    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-home\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-home\") pod \"d5e318ad-55f2-4294-a8c5-10400322462b\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") "
Oct 30 19:47:06.997834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.997442    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-dshm\") pod \"d5e318ad-55f2-4294-a8c5-10400322462b\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") "
Oct 30 19:47:06.997834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.997521    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-tmp-dir\") pod \"d5e318ad-55f2-4294-a8c5-10400322462b\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") "
Oct 30 19:47:06.997834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.997624    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e318ad-55f2-4294-a8c5-10400322462b-kube-api-access-xjm2n\" (UniqueName: \"kubernetes.io/projected/d5e318ad-55f2-4294-a8c5-10400322462b-kube-api-access-xjm2n\") pod \"d5e318ad-55f2-4294-a8c5-10400322462b\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") "
Oct 30 19:47:06.997834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.997629    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-home" pod "d5e318ad-55f2-4294-a8c5-10400322462b" (UID: "d5e318ad-55f2-4294-a8c5-10400322462b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:06.997834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.997656    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e318ad-55f2-4294-a8c5-10400322462b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e318ad-55f2-4294-a8c5-10400322462b-tls-certs\") pod \"d5e318ad-55f2-4294-a8c5-10400322462b\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") "
Oct 30 19:47:06.997834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.997828    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-tmp-dir" pod "d5e318ad-55f2-4294-a8c5-10400322462b" (UID: "d5e318ad-55f2-4294-a8c5-10400322462b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:06.998219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.997873    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-model-cache\") pod \"d5e318ad-55f2-4294-a8c5-10400322462b\" (UID: \"d5e318ad-55f2-4294-a8c5-10400322462b\") "
Oct 30 19:47:06.998219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:06.998211    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-model-cache" pod "d5e318ad-55f2-4294-a8c5-10400322462b" (UID: "d5e318ad-55f2-4294-a8c5-10400322462b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.000671 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.000641    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.000809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.000770    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-dshm" pod "d5e318ad-55f2-4294-a8c5-10400322462b" (UID: "d5e318ad-55f2-4294-a8c5-10400322462b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.000809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.000789    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e318ad-55f2-4294-a8c5-10400322462b-kube-api-access-xjm2n" pod "d5e318ad-55f2-4294-a8c5-10400322462b" (UID: "d5e318ad-55f2-4294-a8c5-10400322462b"). InnerVolumeSpecName "kube-api-access-xjm2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:07.000939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.000822    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b71be04-6fbe-4637-a36b-ff16915906f2-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.000939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.000838    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.000939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.000852    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b71be04-6fbe-4637-a36b-ff16915906f2-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.000939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.000867    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5z5g\" (UniqueName: \"kubernetes.io/projected/3b71be04-6fbe-4637-a36b-ff16915906f2-kube-api-access-z5z5g\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.000939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.000882    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.002617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.002463    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e318ad-55f2-4294-a8c5-10400322462b-tls-certs" pod "d5e318ad-55f2-4294-a8c5-10400322462b" (UID: "d5e318ad-55f2-4294-a8c5-10400322462b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:07.087522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.087483    6777 generic.go:356] "Generic (PLEG): container finished" podID="d5e318ad-55f2-4294-a8c5-10400322462b" containerID="27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6" exitCode=0
Oct 30 19:47:07.087739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.087538    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"
Oct 30 19:47:07.087739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.087584    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" event={"ID":"d5e318ad-55f2-4294-a8c5-10400322462b","Type":"ContainerDied","Data":"27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6"}
Oct 30 19:47:07.087739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.087714    6777 scope.go:122] "RemoveContainer" containerID="27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6"
Oct 30 19:47:07.088148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.088112    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx" event={"ID":"d5e318ad-55f2-4294-a8c5-10400322462b","Type":"ContainerDied","Data":"f128b3fba517388c63c209af9eec5dde5b4d3cea5e61b43d8a9d3fa5fe8fb05d"}
Oct 30 19:47:07.090840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.089879    6777 generic.go:356] "Generic (PLEG): container finished" podID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerID="dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651" exitCode=0
Oct 30 19:47:07.090840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.090134    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" event={"ID":"3b71be04-6fbe-4637-a36b-ff16915906f2","Type":"ContainerDied","Data":"dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651"}
Oct 30 19:47:07.090840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.090173    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc" event={"ID":"3b71be04-6fbe-4637-a36b-ff16915906f2","Type":"ContainerDied","Data":"5a88dc6d16228775c94235a855a8a566f51ce56e25fc1e0f9573a48210facf07"}
Oct 30 19:47:07.090840 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.090320    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"
Oct 30 19:47:07.106882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.106851    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e318ad-55f2-4294-a8c5-10400322462b-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.107139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.107121    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjm2n\" (UniqueName: \"kubernetes.io/projected/d5e318ad-55f2-4294-a8c5-10400322462b-kube-api-access-xjm2n\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.107252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.107237    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e318ad-55f2-4294-a8c5-10400322462b-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.111440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.111270    6777 scope.go:122] "RemoveContainer" containerID="27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6"
Oct 30 19:47:07.111878 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:07.111845    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6\": container with ID starting with 27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6 not found: ID does not exist" containerID="27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6"
Oct 30 19:47:07.112185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.111887    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6"} err="failed to get container status \"27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6\": rpc error: code = NotFound desc = could not find container \"27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6\": container with ID starting with 27d2b6b16df96a5e1754d16ca8551029cdc61b68c82a872cffbb37e423cb7db6 not found: ID does not exist"
Oct 30 19:47:07.112185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.111910    6777 scope.go:122] "RemoveContainer" containerID="dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651"
Oct 30 19:47:07.124608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.124581    6777 scope.go:122] "RemoveContainer" containerID="dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651"
Oct 30 19:47:07.125035 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:07.125009    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651\": container with ID starting with dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651 not found: ID does not exist" containerID="dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651"
Oct 30 19:47:07.125250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.125222    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651"} err="failed to get container status \"dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651\": rpc error: code = NotFound desc = could not find container \"dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651\": container with ID starting with dcf36af45081fa3965ef12f33a335df6ab03cdfa7c22cd2daa5c5bdf55773651 not found: ID does not exist"
Oct 30 19:47:07.129907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.129882    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"]
Oct 30 19:47:07.133597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.133570    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/premium-w0-ad23c7-kserve-959f5cf5d-qkzfx"]
Oct 30 19:47:07.145776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.145743    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"]
Oct 30 19:47:07.149570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.149544    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-2-w0-ad23c7-kserve-7d9db946c8-stwfc"]
Oct 30 19:47:07.762040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.761999    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"]
Oct 30 19:47:07.762353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.762323    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy" containerID="cri-o://9fa629c139054aa591fa11dcdf878f358253cf52483fa9133d19dd9105eb2738" gracePeriod=30
Oct 30 19:47:07.773753 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.773712    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"]
Oct 30 19:47:07.773997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.773970    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy" containerID="cri-o://387fb8a0f1959c3f51aa6b0defa3fcb86ba01187d59b2f4ea0d7726b4eddaffb" gracePeriod=30
Oct 30 19:47:07.778478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.778447    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"]
Oct 30 19:47:07.778679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:07.778656    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy" containerID="cri-o://de76158be39008f19c9d4a4a7f319624301b521c2f9dfeb43fb9724ddd567390" gracePeriod=30
Oct 30 19:47:08.111140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.111107    6777 generic.go:356] "Generic (PLEG): container finished" podID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerID="7b5e16a3718a23d71b1abff2557a8e02da2355d2075c9216d0b3f95edd55371a" exitCode=0
Oct 30 19:47:08.118022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.117989    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" path="/var/lib/kubelet/pods/3b71be04-6fbe-4637-a36b-ff16915906f2/volumes"
Oct 30 19:47:08.118986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.118964    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" path="/var/lib/kubelet/pods/d5e318ad-55f2-4294-a8c5-10400322462b/volumes"
Oct 30 19:47:08.119385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.119365    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" event={"ID":"fb7327b5-03ab-49fa-89e7-08df93bf233d","Type":"ContainerDied","Data":"7b5e16a3718a23d71b1abff2557a8e02da2355d2075c9216d0b3f95edd55371a"}
Oct 30 19:47:08.119519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.119495    6777 scope.go:122] "RemoveContainer" containerID="a765f43cfc2b6c6774889b9623c1125ec653f20553ba0519cfe955df3aa961c0"
Oct 30 19:47:08.256785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.256755    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:47:08.333343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.333307    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-token\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-token\") pod \"fb7327b5-03ab-49fa-89e7-08df93bf233d\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") "
Oct 30 19:47:08.333532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.333420    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-envoy\") pod \"fb7327b5-03ab-49fa-89e7-08df93bf233d\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") "
Oct 30 19:47:08.333532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.333453    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-podinfo\") pod \"fb7327b5-03ab-49fa-89e7-08df93bf233d\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") "
Oct 30 19:47:08.333532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.333484    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-credential-socket\") pod \"fb7327b5-03ab-49fa-89e7-08df93bf233d\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") "
Oct 30 19:47:08.333706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.333547    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-socket\") pod \"fb7327b5-03ab-49fa-89e7-08df93bf233d\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") "
Oct 30 19:47:08.333706 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.333615    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-certs\") pod \"fb7327b5-03ab-49fa-89e7-08df93bf233d\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") "
Oct 30 19:47:08.333778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.333726    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7327b5-03ab-49fa-89e7-08df93bf233d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7327b5-03ab-49fa-89e7-08df93bf233d-istiod-ca-cert\") pod \"fb7327b5-03ab-49fa-89e7-08df93bf233d\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") "
Oct 30 19:47:08.333831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.333818    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-kube-api-access-r5c54\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-kube-api-access-r5c54\") pod \"fb7327b5-03ab-49fa-89e7-08df93bf233d\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") "
Oct 30 19:47:08.333905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.333875    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-data\") pod \"fb7327b5-03ab-49fa-89e7-08df93bf233d\" (UID: \"fb7327b5-03ab-49fa-89e7-08df93bf233d\") "
Oct 30 19:47:08.335150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.334918    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-socket" pod "fb7327b5-03ab-49fa-89e7-08df93bf233d" (UID: "fb7327b5-03ab-49fa-89e7-08df93bf233d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.335582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.335550    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-certs" pod "fb7327b5-03ab-49fa-89e7-08df93bf233d" (UID: "fb7327b5-03ab-49fa-89e7-08df93bf233d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.336559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.336519    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-credential-socket" pod "fb7327b5-03ab-49fa-89e7-08df93bf233d" (UID: "fb7327b5-03ab-49fa-89e7-08df93bf233d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.336692 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.336647    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-token" pod "fb7327b5-03ab-49fa-89e7-08df93bf233d" (UID: "fb7327b5-03ab-49fa-89e7-08df93bf233d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:08.336812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.336783    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-podinfo" pod "fb7327b5-03ab-49fa-89e7-08df93bf233d" (UID: "fb7327b5-03ab-49fa-89e7-08df93bf233d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:08.341124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.337834    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7327b5-03ab-49fa-89e7-08df93bf233d-istiod-ca-cert" pod "fb7327b5-03ab-49fa-89e7-08df93bf233d" (UID: "fb7327b5-03ab-49fa-89e7-08df93bf233d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:08.341124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.337866    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-kube-api-access-r5c54" pod "fb7327b5-03ab-49fa-89e7-08df93bf233d" (UID: "fb7327b5-03ab-49fa-89e7-08df93bf233d"). InnerVolumeSpecName "kube-api-access-r5c54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:08.341124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.338045    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-data" pod "fb7327b5-03ab-49fa-89e7-08df93bf233d" (UID: "fb7327b5-03ab-49fa-89e7-08df93bf233d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.341124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.338248    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-envoy" pod "fb7327b5-03ab-49fa-89e7-08df93bf233d" (UID: "fb7327b5-03ab-49fa-89e7-08df93bf233d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.436616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.436578    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.436616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.436609    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.436616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.436618    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.436616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.436628    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7327b5-03ab-49fa-89e7-08df93bf233d-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.437021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.436638    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.437021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.436647    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.437021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.436657    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7327b5-03ab-49fa-89e7-08df93bf233d-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.437021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.436666    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7327b5-03ab-49fa-89e7-08df93bf233d-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.437021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:08.436674    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c54\" (UniqueName: \"kubernetes.io/projected/fb7327b5-03ab-49fa-89e7-08df93bf233d-kube-api-access-r5c54\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:09.123758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:09.123726    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"
Oct 30 19:47:09.124322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:09.123768    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl" event={"ID":"fb7327b5-03ab-49fa-89e7-08df93bf233d","Type":"ContainerDied","Data":"463a82eb76f340a947c74160ebcf63b10511651e8344ada10e5f23e480d09690"}
Oct 30 19:47:09.124322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:09.123832    6777 scope.go:122] "RemoveContainer" containerID="7b5e16a3718a23d71b1abff2557a8e02da2355d2075c9216d0b3f95edd55371a"
Oct 30 19:47:09.150660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:09.150624    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"]
Oct 30 19:47:09.158465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:09.158433    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-rwpzl"]
Oct 30 19:47:10.114723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:10.114684    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" path="/var/lib/kubelet/pods/fb7327b5-03ab-49fa-89e7-08df93bf233d/volumes"
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.165449    6777 generic.go:356] "Generic (PLEG): container finished" podID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerID="387fb8a0f1959c3f51aa6b0defa3fcb86ba01187d59b2f4ea0d7726b4eddaffb" exitCode=0
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.165610    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" event={"ID":"ed339902-7cac-4388-b69e-f1768cfe44e0","Type":"ContainerDied","Data":"387fb8a0f1959c3f51aa6b0defa3fcb86ba01187d59b2f4ea0d7726b4eddaffb"}
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.168479    6777 generic.go:356] "Generic (PLEG): container finished" podID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerID="9fa629c139054aa591fa11dcdf878f358253cf52483fa9133d19dd9105eb2738" exitCode=0
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.168563    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" event={"ID":"fcfd0a26-981e-4313-b6d4-66dbf0107982","Type":"ContainerDied","Data":"9fa629c139054aa591fa11dcdf878f358253cf52483fa9133d19dd9105eb2738"}
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.168612    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x" event={"ID":"fcfd0a26-981e-4313-b6d4-66dbf0107982","Type":"ContainerDied","Data":"10de4d105a405531de73c0aac966281326c262c343f4b73c195de1fe9159dbff"}
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.168702    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10de4d105a405531de73c0aac966281326c262c343f4b73c195de1fe9159dbff"
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.168721    6777 scope.go:122] "RemoveContainer" containerID="206c54088f0a9d30ba7d4a55053462985cacc6ab41e5035ae4175334ad9b16e7"
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.171427    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.172891    6777 generic.go:356] "Generic (PLEG): container finished" podID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerID="de76158be39008f19c9d4a4a7f319624301b521c2f9dfeb43fb9724ddd567390" exitCode=0
Oct 30 19:47:13.181988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.172958    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" event={"ID":"1eb75217-842f-45f8-8e88-61453b0f64ed","Type":"ContainerDied","Data":"de76158be39008f19c9d4a4a7f319624301b521c2f9dfeb43fb9724ddd567390"}
Oct 30 19:47:13.239904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.239873    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:47:13.251092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.251033    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-podinfo\") pod \"fcfd0a26-981e-4313-b6d4-66dbf0107982\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") "
Oct 30 19:47:13.251268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.251212    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-envoy\") pod \"fcfd0a26-981e-4313-b6d4-66dbf0107982\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") "
Oct 30 19:47:13.251268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.251246    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-data\") pod \"fcfd0a26-981e-4313-b6d4-66dbf0107982\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") "
Oct 30 19:47:13.251402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.251301    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-token\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-token\") pod \"fcfd0a26-981e-4313-b6d4-66dbf0107982\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") "
Oct 30 19:47:13.251402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.251384    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-kube-api-access-zf87c\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-kube-api-access-zf87c\") pod \"fcfd0a26-981e-4313-b6d4-66dbf0107982\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") "
Oct 30 19:47:13.251511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.251460    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfd0a26-981e-4313-b6d4-66dbf0107982-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcfd0a26-981e-4313-b6d4-66dbf0107982-istiod-ca-cert\") pod \"fcfd0a26-981e-4313-b6d4-66dbf0107982\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") "
Oct 30 19:47:13.251569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.251557    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-credential-socket\") pod \"fcfd0a26-981e-4313-b6d4-66dbf0107982\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") "
Oct 30 19:47:13.251652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.251632    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-certs\") pod \"fcfd0a26-981e-4313-b6d4-66dbf0107982\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") "
Oct 30 19:47:13.251795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.251775    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-socket\") pod \"fcfd0a26-981e-4313-b6d4-66dbf0107982\" (UID: \"fcfd0a26-981e-4313-b6d4-66dbf0107982\") "
Oct 30 19:47:13.252834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.252765    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:47:13.253501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.253470    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-certs" pod "fcfd0a26-981e-4313-b6d4-66dbf0107982" (UID: "fcfd0a26-981e-4313-b6d4-66dbf0107982"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.254809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.254089    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfd0a26-981e-4313-b6d4-66dbf0107982-istiod-ca-cert" pod "fcfd0a26-981e-4313-b6d4-66dbf0107982" (UID: "fcfd0a26-981e-4313-b6d4-66dbf0107982"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:13.254809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.254093    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-data" pod "fcfd0a26-981e-4313-b6d4-66dbf0107982" (UID: "fcfd0a26-981e-4313-b6d4-66dbf0107982"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.254809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.254165    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-credential-socket" pod "fcfd0a26-981e-4313-b6d4-66dbf0107982" (UID: "fcfd0a26-981e-4313-b6d4-66dbf0107982"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.254809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.254451    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-socket" pod "fcfd0a26-981e-4313-b6d4-66dbf0107982" (UID: "fcfd0a26-981e-4313-b6d4-66dbf0107982"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.257448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.257421    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-token" pod "fcfd0a26-981e-4313-b6d4-66dbf0107982" (UID: "fcfd0a26-981e-4313-b6d4-66dbf0107982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.257545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.257477    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-podinfo" pod "fcfd0a26-981e-4313-b6d4-66dbf0107982" (UID: "fcfd0a26-981e-4313-b6d4-66dbf0107982"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:13.257545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.257525    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-envoy" pod "fcfd0a26-981e-4313-b6d4-66dbf0107982" (UID: "fcfd0a26-981e-4313-b6d4-66dbf0107982"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.259408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.259379    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-kube-api-access-zf87c" pod "fcfd0a26-981e-4313-b6d4-66dbf0107982" (UID: "fcfd0a26-981e-4313-b6d4-66dbf0107982"). InnerVolumeSpecName "kube-api-access-zf87c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.354127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354020    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-podinfo\") pod \"1eb75217-842f-45f8-8e88-61453b0f64ed\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") "
Oct 30 19:47:13.354305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354184    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-data\") pod \"1eb75217-842f-45f8-8e88-61453b0f64ed\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") "
Oct 30 19:47:13.354305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354258    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-data\") pod \"ed339902-7cac-4388-b69e-f1768cfe44e0\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") "
Oct 30 19:47:13.354447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354285    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-certs\") pod \"ed339902-7cac-4388-b69e-f1768cfe44e0\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") "
Oct 30 19:47:13.354447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354372    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-kube-api-access-5z4dq\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-kube-api-access-5z4dq\") pod \"1eb75217-842f-45f8-8e88-61453b0f64ed\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") "
Oct 30 19:47:13.354447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354414    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-podinfo\") pod \"ed339902-7cac-4388-b69e-f1768cfe44e0\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") "
Oct 30 19:47:13.354676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354452    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-credential-socket\") pod \"ed339902-7cac-4388-b69e-f1768cfe44e0\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") "
Oct 30 19:47:13.354676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354523    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-token\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-token\") pod \"1eb75217-842f-45f8-8e88-61453b0f64ed\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") "
Oct 30 19:47:13.354676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354560    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb75217-842f-45f8-8e88-61453b0f64ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb75217-842f-45f8-8e88-61453b0f64ed-istiod-ca-cert\") pod \"1eb75217-842f-45f8-8e88-61453b0f64ed\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") "
Oct 30 19:47:13.354676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354621    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-data" pod "ed339902-7cac-4388-b69e-f1768cfe44e0" (UID: "ed339902-7cac-4388-b69e-f1768cfe44e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.354897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354678    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-kube-api-access-69twq\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-kube-api-access-69twq\") pod \"ed339902-7cac-4388-b69e-f1768cfe44e0\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") "
Oct 30 19:47:13.354897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354720    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-envoy\") pod \"1eb75217-842f-45f8-8e88-61453b0f64ed\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") "
Oct 30 19:47:13.354897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354766    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-socket\") pod \"ed339902-7cac-4388-b69e-f1768cfe44e0\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") "
Oct 30 19:47:13.354897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354799    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-envoy\") pod \"ed339902-7cac-4388-b69e-f1768cfe44e0\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") "
Oct 30 19:47:13.354897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354855    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-credential-socket\") pod \"1eb75217-842f-45f8-8e88-61453b0f64ed\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") "
Oct 30 19:47:13.355219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354910    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-data" pod "1eb75217-842f-45f8-8e88-61453b0f64ed" (UID: "1eb75217-842f-45f8-8e88-61453b0f64ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.355219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354919    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-credential-socket" pod "ed339902-7cac-4388-b69e-f1768cfe44e0" (UID: "ed339902-7cac-4388-b69e-f1768cfe44e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.355219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.354923    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-certs\") pod \"1eb75217-842f-45f8-8e88-61453b0f64ed\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") "
Oct 30 19:47:13.355219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.355002    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-socket\") pod \"1eb75217-842f-45f8-8e88-61453b0f64ed\" (UID: \"1eb75217-842f-45f8-8e88-61453b0f64ed\") "
Oct 30 19:47:13.355219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.355194    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-token\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-token\") pod \"ed339902-7cac-4388-b69e-f1768cfe44e0\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") "
Oct 30 19:47:13.355219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.355198    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-certs" pod "1eb75217-842f-45f8-8e88-61453b0f64ed" (UID: "1eb75217-842f-45f8-8e88-61453b0f64ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.355601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.355232    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed339902-7cac-4388-b69e-f1768cfe44e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed339902-7cac-4388-b69e-f1768cfe44e0-istiod-ca-cert\") pod \"ed339902-7cac-4388-b69e-f1768cfe44e0\" (UID: \"ed339902-7cac-4388-b69e-f1768cfe44e0\") "
Oct 30 19:47:13.355601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.355538    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-certs" pod "ed339902-7cac-4388-b69e-f1768cfe44e0" (UID: "ed339902-7cac-4388-b69e-f1768cfe44e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.355779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.355756    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-socket" pod "1eb75217-842f-45f8-8e88-61453b0f64ed" (UID: "1eb75217-842f-45f8-8e88-61453b0f64ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356160    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-socket" pod "ed339902-7cac-4388-b69e-f1768cfe44e0" (UID: "ed339902-7cac-4388-b69e-f1768cfe44e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356213    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb75217-842f-45f8-8e88-61453b0f64ed-istiod-ca-cert" pod "1eb75217-842f-45f8-8e88-61453b0f64ed" (UID: "1eb75217-842f-45f8-8e88-61453b0f64ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356371    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcfd0a26-981e-4313-b6d4-66dbf0107982-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356391    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356410    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356424    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356438    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356453    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356467    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356480    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.356485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356494    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.357325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356507    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.357325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356520    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.357325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356534    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.357325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356546    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.357325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356559    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.357325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356573    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.357325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356634    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf87c\" (UniqueName: \"kubernetes.io/projected/fcfd0a26-981e-4313-b6d4-66dbf0107982-kube-api-access-zf87c\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.357325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356934    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed339902-7cac-4388-b69e-f1768cfe44e0-istiod-ca-cert" pod "ed339902-7cac-4388-b69e-f1768cfe44e0" (UID: "ed339902-7cac-4388-b69e-f1768cfe44e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:13.357325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.356997    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-credential-socket" pod "1eb75217-842f-45f8-8e88-61453b0f64ed" (UID: "1eb75217-842f-45f8-8e88-61453b0f64ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.357797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.357676    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-kube-api-access-5z4dq" pod "1eb75217-842f-45f8-8e88-61453b0f64ed" (UID: "1eb75217-842f-45f8-8e88-61453b0f64ed"). InnerVolumeSpecName "kube-api-access-5z4dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.357797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.357705    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-token" pod "1eb75217-842f-45f8-8e88-61453b0f64ed" (UID: "1eb75217-842f-45f8-8e88-61453b0f64ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.358525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.358486    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-podinfo" pod "1eb75217-842f-45f8-8e88-61453b0f64ed" (UID: "1eb75217-842f-45f8-8e88-61453b0f64ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:13.360825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.360801    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-kube-api-access-69twq" pod "ed339902-7cac-4388-b69e-f1768cfe44e0" (UID: "ed339902-7cac-4388-b69e-f1768cfe44e0"). InnerVolumeSpecName "kube-api-access-69twq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.364564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.364524    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-podinfo" pod "ed339902-7cac-4388-b69e-f1768cfe44e0" (UID: "ed339902-7cac-4388-b69e-f1768cfe44e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:13.364564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.364538    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-envoy" pod "ed339902-7cac-4388-b69e-f1768cfe44e0" (UID: "ed339902-7cac-4388-b69e-f1768cfe44e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.364702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.364572    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-token" pod "ed339902-7cac-4388-b69e-f1768cfe44e0" (UID: "ed339902-7cac-4388-b69e-f1768cfe44e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.366042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.366013    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-envoy" pod "1eb75217-842f-45f8-8e88-61453b0f64ed" (UID: "1eb75217-842f-45f8-8e88-61453b0f64ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.458153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458118    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458152    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458164    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb75217-842f-45f8-8e88-61453b0f64ed-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458176    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458184    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed339902-7cac-4388-b69e-f1768cfe44e0-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458192    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458202    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z4dq\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-kube-api-access-5z4dq\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458211    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed339902-7cac-4388-b69e-f1768cfe44e0-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458220    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb75217-842f-45f8-8e88-61453b0f64ed-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458228    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb75217-842f-45f8-8e88-61453b0f64ed-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.458358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:13.458237    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69twq\" (UniqueName: \"kubernetes.io/projected/ed339902-7cac-4388-b69e-f1768cfe44e0-kube-api-access-69twq\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.748828 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:13.748725    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:47:14.188683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.188654    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"
Oct 30 19:47:14.190269 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.190242    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"
Oct 30 19:47:14.190383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.190264    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr" event={"ID":"1eb75217-842f-45f8-8e88-61453b0f64ed","Type":"ContainerDied","Data":"b77999406f6402fe6284873c964f24b45d0878e98f5827f4c89b21b52d86d421"}
Oct 30 19:47:14.190383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.190323    6777 scope.go:122] "RemoveContainer" containerID="de76158be39008f19c9d4a4a7f319624301b521c2f9dfeb43fb9724ddd567390"
Oct 30 19:47:14.192183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.192160    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf" event={"ID":"ed339902-7cac-4388-b69e-f1768cfe44e0","Type":"ContainerDied","Data":"a4689204adbf35c59b4fa0d4037b8e979d5ddcf85866556fcace1e97379e69d9"}
Oct 30 19:47:14.192333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.192260    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"
Oct 30 19:47:14.201695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.201674    6777 scope.go:122] "RemoveContainer" containerID="387fb8a0f1959c3f51aa6b0defa3fcb86ba01187d59b2f4ea0d7726b4eddaffb"
Oct 30 19:47:14.253239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.253206    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"]
Oct 30 19:47:14.283740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.283697    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-kcs7x"]
Oct 30 19:47:14.300592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.300555    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"]
Oct 30 19:47:14.303838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.303812    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-hl9lr"]
Oct 30 19:47:14.324940 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.324897    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"]
Oct 30 19:47:14.330565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.330532    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-fv8pf"]
Oct 30 19:47:14.509614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:14.509526    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896" podUID="1f430439-89da-4b5a-9c8e-4bb5997ad43d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:15.580105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:15.579231    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"]
Oct 30 19:47:15.580105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:15.579449    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc" podUID="c315da14-f28a-4be1-a90d-ff186090fb9e" containerName="maas-api" containerID="cri-o://1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f" gracePeriod=30
Oct 30 19:47:16.077633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.077606    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:47:16.118717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.118683    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" path="/var/lib/kubelet/pods/1eb75217-842f-45f8-8e88-61453b0f64ed/volumes"
Oct 30 19:47:16.119355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.119338    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" path="/var/lib/kubelet/pods/ed339902-7cac-4388-b69e-f1768cfe44e0/volumes"
Oct 30 19:47:16.119818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.119804    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" path="/var/lib/kubelet/pods/fcfd0a26-981e-4313-b6d4-66dbf0107982/volumes"
Oct 30 19:47:16.156139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.156042    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c315da14-f28a-4be1-a90d-ff186090fb9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c315da14-f28a-4be1-a90d-ff186090fb9e-openshift-service-ca\") pod \"c315da14-f28a-4be1-a90d-ff186090fb9e\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") "
Oct 30 19:47:16.156326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.156179    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c315da14-f28a-4be1-a90d-ff186090fb9e-kube-api-access-dbkz4\" (UniqueName: \"kubernetes.io/projected/c315da14-f28a-4be1-a90d-ff186090fb9e-kube-api-access-dbkz4\") pod \"c315da14-f28a-4be1-a90d-ff186090fb9e\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") "
Oct 30 19:47:16.156326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.156240    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls\") pod \"c315da14-f28a-4be1-a90d-ff186090fb9e\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") "
Oct 30 19:47:16.156326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.156305    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-metrics-certs\") pod \"c315da14-f28a-4be1-a90d-ff186090fb9e\" (UID: \"c315da14-f28a-4be1-a90d-ff186090fb9e\") "
Oct 30 19:47:16.157151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.157118    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c315da14-f28a-4be1-a90d-ff186090fb9e-openshift-service-ca" pod "c315da14-f28a-4be1-a90d-ff186090fb9e" (UID: "c315da14-f28a-4be1-a90d-ff186090fb9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:16.159025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.158988    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls" pod "c315da14-f28a-4be1-a90d-ff186090fb9e" (UID: "c315da14-f28a-4be1-a90d-ff186090fb9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:16.159251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.159029    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-metrics-certs" pod "c315da14-f28a-4be1-a90d-ff186090fb9e" (UID: "c315da14-f28a-4be1-a90d-ff186090fb9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:16.159251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.159156    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c315da14-f28a-4be1-a90d-ff186090fb9e-kube-api-access-dbkz4" pod "c315da14-f28a-4be1-a90d-ff186090fb9e" (UID: "c315da14-f28a-4be1-a90d-ff186090fb9e"). InnerVolumeSpecName "kube-api-access-dbkz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:16.223143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.215197    6777 generic.go:356] "Generic (PLEG): container finished" podID="c315da14-f28a-4be1-a90d-ff186090fb9e" containerID="1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f" exitCode=0
Oct 30 19:47:16.223143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.215508    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"
Oct 30 19:47:16.223143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.215516    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc" event={"ID":"c315da14-f28a-4be1-a90d-ff186090fb9e","Type":"ContainerDied","Data":"1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f"}
Oct 30 19:47:16.229203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.226012    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc" event={"ID":"c315da14-f28a-4be1-a90d-ff186090fb9e","Type":"ContainerDied","Data":"471b155faf8ce189a3caeb115d62b2a04307bc5ea2ee84d2133e2be6dbde4dba"}
Oct 30 19:47:16.229203 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.226153    6777 scope.go:122] "RemoveContainer" containerID="1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f"
Oct 30 19:47:16.256379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.256336    6777 scope.go:122] "RemoveContainer" containerID="1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f"
Oct 30 19:47:16.256998 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:16.256962    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f\": container with ID starting with 1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f not found: ID does not exist" containerID="1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f"
Oct 30 19:47:16.257219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.257014    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f"} err="failed to get container status \"1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f\": rpc error: code = NotFound desc = could not find container \"1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f\": container with ID starting with 1363c22e5ca86379b90eec2f0734428255e9968eafe5dc62faa315811d070d6f not found: ID does not exist"
Oct 30 19:47:16.260864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.260350    6777 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c315da14-f28a-4be1-a90d-ff186090fb9e-openshift-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:16.260864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.260385    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkz4\" (UniqueName: \"kubernetes.io/projected/c315da14-f28a-4be1-a90d-ff186090fb9e-kube-api-access-dbkz4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:16.260864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.260400    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-maas-api-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:16.260864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.260408    6777 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c315da14-f28a-4be1-a90d-ff186090fb9e-metrics-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:16.279680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.279647    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"]
Oct 30 19:47:16.285922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:16.285880    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ad23c7-89fcb8d-z9phc"]
Oct 30 19:47:17.092822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:17.092789    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:17.093286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:17.092862    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:18.116646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:18.116613    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c315da14-f28a-4be1-a90d-ff186090fb9e" path="/var/lib/kubelet/pods/c315da14-f28a-4be1-a90d-ff186090fb9e/volumes"
Oct 30 19:47:22.117120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.117091    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:47:22.196787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.196750    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f430439-89da-4b5a-9c8e-4bb5997ad43d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f430439-89da-4b5a-9c8e-4bb5997ad43d-plugins-config-volume\") pod \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\" (UID: \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\") "
Oct 30 19:47:22.196984 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.196830    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f430439-89da-4b5a-9c8e-4bb5997ad43d-kube-api-access-mdg2g\" (UniqueName: \"kubernetes.io/projected/1f430439-89da-4b5a-9c8e-4bb5997ad43d-kube-api-access-mdg2g\") pod \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\" (UID: \"1f430439-89da-4b5a-9c8e-4bb5997ad43d\") "
Oct 30 19:47:22.197286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.197259    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f430439-89da-4b5a-9c8e-4bb5997ad43d-plugins-config-volume" pod "1f430439-89da-4b5a-9c8e-4bb5997ad43d" (UID: "1f430439-89da-4b5a-9c8e-4bb5997ad43d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:22.197952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.197927    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f430439-89da-4b5a-9c8e-4bb5997ad43d-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:22.199350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.199319    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f430439-89da-4b5a-9c8e-4bb5997ad43d-kube-api-access-mdg2g" pod "1f430439-89da-4b5a-9c8e-4bb5997ad43d" (UID: "1f430439-89da-4b5a-9c8e-4bb5997ad43d"). InnerVolumeSpecName "kube-api-access-mdg2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:22.267536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.267501    6777 generic.go:356] "Generic (PLEG): container finished" podID="1f430439-89da-4b5a-9c8e-4bb5997ad43d" containerID="c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2" exitCode=0
Oct 30 19:47:22.267727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.267550    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"
Oct 30 19:47:22.267727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.267586    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896" event={"ID":"1f430439-89da-4b5a-9c8e-4bb5997ad43d","Type":"ContainerDied","Data":"c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2"}
Oct 30 19:47:22.267727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.267649    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896" event={"ID":"1f430439-89da-4b5a-9c8e-4bb5997ad43d","Type":"ContainerDied","Data":"1f3a579a24aeb5b2a091abffc3966664956314ecb3d95d3a1a617018cc860ad0"}
Oct 30 19:47:22.267727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.267677    6777 scope.go:122] "RemoveContainer" containerID="c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2"
Oct 30 19:47:22.279482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.279463    6777 scope.go:122] "RemoveContainer" containerID="c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2"
Oct 30 19:47:22.279847 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:22.279824    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2\": container with ID starting with c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2 not found: ID does not exist" containerID="c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2"
Oct 30 19:47:22.279915 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.279856    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2"} err="failed to get container status \"c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2\": rpc error: code = NotFound desc = could not find container \"c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2\": container with ID starting with c42beba813edfa3c629e0fa036196c3a4fe379f4fbb39080516546a8332cf7d2 not found: ID does not exist"
Oct 30 19:47:22.299179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.299148    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdg2g\" (UniqueName: \"kubernetes.io/projected/1f430439-89da-4b5a-9c8e-4bb5997ad43d-kube-api-access-mdg2g\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:22.309390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.309353    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"]
Oct 30 19:47:22.312349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:22.312321    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ad23c7-777db74db7-v2896"]
Oct 30 19:47:24.115375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:24.115340    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f430439-89da-4b5a-9c8e-4bb5997ad43d" path="/var/lib/kubelet/pods/1f430439-89da-4b5a-9c8e-4bb5997ad43d/volumes"
Oct 30 19:47:24.601723 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:24.601685    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"]
Oct 30 19:47:24.601998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:24.601948    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy" containerID="cri-o://1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486" gracePeriod=30
Oct 30 19:47:25.203138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:25.203099    6777 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:47:25.203526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:25.203166    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:47:26.234002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.233964    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"]
Oct 30 19:47:26.234923 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.234897    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c315da14-f28a-4be1-a90d-ff186090fb9e" containerName="maas-api"
Oct 30 19:47:26.235036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.234924    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c315da14-f28a-4be1-a90d-ff186090fb9e" containerName="maas-api"
Oct 30 19:47:26.235036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.234938    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy"
Oct 30 19:47:26.235036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.234945    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy"
Oct 30 19:47:26.235036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.234970    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy"
Oct 30 19:47:26.235036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.234977    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy"
Oct 30 19:47:26.235036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.234988    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy"
Oct 30 19:47:26.235036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.234995    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy"
Oct 30 19:47:26.235036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235025    6777 cpu_manager.go:414] "containerMap: removing container" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main"
Oct 30 19:47:26.235036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235034    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235080    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235088    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235113    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1f430439-89da-4b5a-9c8e-4bb5997ad43d" containerName="payload-pre-processing"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235120    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f430439-89da-4b5a-9c8e-4bb5997ad43d" containerName="payload-pre-processing"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235133    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235139    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235354    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed339902-7cac-4388-b69e-f1768cfe44e0" containerName="istio-proxy"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235372    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f430439-89da-4b5a-9c8e-4bb5997ad43d" containerName="payload-pre-processing"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235382    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e318ad-55f2-4294-a8c5-10400322462b" containerName="main"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235396    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235408    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235422    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c315da14-f28a-4be1-a90d-ff186090fb9e" containerName="maas-api"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235432    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235439    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb75217-842f-45f8-8e88-61453b0f64ed" containerName="istio-proxy"
Oct 30 19:47:26.235572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.235449    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b71be04-6fbe-4637-a36b-ff16915906f2" containerName="main"
Oct 30 19:47:26.239662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.239640    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:26.250328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.250113    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"]
Oct 30 19:47:26.250986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.250956    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-be6b74\""
Oct 30 19:47:26.307788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.307754    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b018b2c-5358-4414-b836-baa46bc655e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj\" (UID: \"0b018b2c-5358-4414-b836-baa46bc655e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:26.307959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.307802    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lj95\" (UniqueName: \"kubernetes.io/projected/0b018b2c-5358-4414-b836-baa46bc655e3-kube-api-access-2lj95\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj\" (UID: \"0b018b2c-5358-4414-b836-baa46bc655e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:26.428109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.410228    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lj95\" (UniqueName: \"kubernetes.io/projected/0b018b2c-5358-4414-b836-baa46bc655e3-kube-api-access-2lj95\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj\" (UID: \"0b018b2c-5358-4414-b836-baa46bc655e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:26.428109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.410645    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b018b2c-5358-4414-b836-baa46bc655e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj\" (UID: \"0b018b2c-5358-4414-b836-baa46bc655e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:26.428109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.411646    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b018b2c-5358-4414-b836-baa46bc655e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj\" (UID: \"0b018b2c-5358-4414-b836-baa46bc655e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:26.459591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.459553    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lj95\" (UniqueName: \"kubernetes.io/projected/0b018b2c-5358-4414-b836-baa46bc655e3-kube-api-access-2lj95\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj\" (UID: \"0b018b2c-5358-4414-b836-baa46bc655e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:26.562157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.559328    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:26.864848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:26.864817    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"]
Oct 30 19:47:26.865232 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:47:26.865210    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b018b2c_5358_4414_b836_baa46bc655e3.slice/crio-1dd50a3094cb0977ca21fbb5b7468e878d1d65401c44046e3dcc8a1efe34e4d0 WatchSource:0}: Error finding container 1dd50a3094cb0977ca21fbb5b7468e878d1d65401c44046e3dcc8a1efe34e4d0: Status 404 returned error can't find the container with id 1dd50a3094cb0977ca21fbb5b7468e878d1d65401c44046e3dcc8a1efe34e4d0
Oct 30 19:47:27.313257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:27.313217    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj" event={"ID":"0b018b2c-5358-4414-b836-baa46bc655e3","Type":"ContainerStarted","Data":"3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6"}
Oct 30 19:47:27.313703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:27.313287    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj" event={"ID":"0b018b2c-5358-4414-b836-baa46bc655e3","Type":"ContainerStarted","Data":"1dd50a3094cb0977ca21fbb5b7468e878d1d65401c44046e3dcc8a1efe34e4d0"}
Oct 30 19:47:27.313703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:27.313315    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:27.336149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:27.336093    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj" podStartSLOduration=1.336078892 podStartE2EDuration="1.336078892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:27.332913517 +0000 UTC m=+2429.329828901" watchObservedRunningTime="2026-10-30 19:47:27.336078892 +0000 UTC m=+2429.332994276"
Oct 30 19:47:29.908154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.908124    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:47:29.980328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980297    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-kube-api-access-zvqlp\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-kube-api-access-zvqlp\") pod \"2467237f-3ab9-424a-9899-37f77850510a\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") "
Oct 30 19:47:29.980609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980345    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-envoy\") pod \"2467237f-3ab9-424a-9899-37f77850510a\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") "
Oct 30 19:47:29.980609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980474    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-credential-socket\") pod \"2467237f-3ab9-424a-9899-37f77850510a\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") "
Oct 30 19:47:29.980609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980526    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-data\") pod \"2467237f-3ab9-424a-9899-37f77850510a\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") "
Oct 30 19:47:29.980609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980569    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-certs\") pod \"2467237f-3ab9-424a-9899-37f77850510a\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") "
Oct 30 19:47:29.980845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980649    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2467237f-3ab9-424a-9899-37f77850510a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467237f-3ab9-424a-9899-37f77850510a-istiod-ca-cert\") pod \"2467237f-3ab9-424a-9899-37f77850510a\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") "
Oct 30 19:47:29.980845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980681    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-socket\") pod \"2467237f-3ab9-424a-9899-37f77850510a\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") "
Oct 30 19:47:29.980845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980762    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-istio-token\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-istio-token\") pod \"2467237f-3ab9-424a-9899-37f77850510a\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") "
Oct 30 19:47:29.980845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980828    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-credential-socket" pod "2467237f-3ab9-424a-9899-37f77850510a" (UID: "2467237f-3ab9-424a-9899-37f77850510a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:29.981044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980839    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-data" pod "2467237f-3ab9-424a-9899-37f77850510a" (UID: "2467237f-3ab9-424a-9899-37f77850510a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:29.981044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980950    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2467237f-3ab9-424a-9899-37f77850510a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467237f-3ab9-424a-9899-37f77850510a-istio-podinfo\") pod \"2467237f-3ab9-424a-9899-37f77850510a\" (UID: \"2467237f-3ab9-424a-9899-37f77850510a\") "
Oct 30 19:47:29.981044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.980996    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-certs" pod "2467237f-3ab9-424a-9899-37f77850510a" (UID: "2467237f-3ab9-424a-9899-37f77850510a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:29.981299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.981119    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-socket" pod "2467237f-3ab9-424a-9899-37f77850510a" (UID: "2467237f-3ab9-424a-9899-37f77850510a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:29.981299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.981229    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2467237f-3ab9-424a-9899-37f77850510a-istiod-ca-cert" pod "2467237f-3ab9-424a-9899-37f77850510a" (UID: "2467237f-3ab9-424a-9899-37f77850510a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:29.981795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.981769    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:29.981899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.981798    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:29.981899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.981814    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:29.981899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.981827    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467237f-3ab9-424a-9899-37f77850510a-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:29.981899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.981839    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:29.983437 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.983397    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-istio-token" pod "2467237f-3ab9-424a-9899-37f77850510a" (UID: "2467237f-3ab9-424a-9899-37f77850510a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:29.983646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.983618    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-envoy" pod "2467237f-3ab9-424a-9899-37f77850510a" (UID: "2467237f-3ab9-424a-9899-37f77850510a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:29.983755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.983646    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-kube-api-access-zvqlp" pod "2467237f-3ab9-424a-9899-37f77850510a" (UID: "2467237f-3ab9-424a-9899-37f77850510a"). InnerVolumeSpecName "kube-api-access-zvqlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:29.983755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:29.983681    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2467237f-3ab9-424a-9899-37f77850510a-istio-podinfo" pod "2467237f-3ab9-424a-9899-37f77850510a" (UID: "2467237f-3ab9-424a-9899-37f77850510a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:30.083427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.083390    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467237f-3ab9-424a-9899-37f77850510a-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.083427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.083418    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqlp\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-kube-api-access-zvqlp\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.083427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.083430    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467237f-3ab9-424a-9899-37f77850510a-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.083427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.083439    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2467237f-3ab9-424a-9899-37f77850510a-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.356330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.356291    6777 generic.go:356] "Generic (PLEG): container finished" podID="2467237f-3ab9-424a-9899-37f77850510a" containerID="1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486" exitCode=0
Oct 30 19:47:30.356525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.356342    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"
Oct 30 19:47:30.356525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.356374    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" event={"ID":"2467237f-3ab9-424a-9899-37f77850510a","Type":"ContainerDied","Data":"1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486"}
Oct 30 19:47:30.356525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.356432    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx" event={"ID":"2467237f-3ab9-424a-9899-37f77850510a","Type":"ContainerDied","Data":"485252d31e84a73b99ecc74a004c408c9ba55c3318ac9b47e704455f5ddb50bc"}
Oct 30 19:47:30.356525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.356454    6777 scope.go:122] "RemoveContainer" containerID="1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486"
Oct 30 19:47:30.367773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.367750    6777 scope.go:122] "RemoveContainer" containerID="fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c"
Oct 30 19:47:30.388977 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.388932    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"]
Oct 30 19:47:30.389271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.389214    6777 scope.go:122] "RemoveContainer" containerID="1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486"
Oct 30 19:47:30.389681 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:30.389654    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486\": container with ID starting with 1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486 not found: ID does not exist" containerID="1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486"
Oct 30 19:47:30.389809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.389688    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486"} err="failed to get container status \"1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486\": rpc error: code = NotFound desc = could not find container \"1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486\": container with ID starting with 1ef5b7b2e775dbd2cb8012b14000e75b96e6cc40656fd6355c04563b9d10b486 not found: ID does not exist"
Oct 30 19:47:30.389809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.389714    6777 scope.go:122] "RemoveContainer" containerID="fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c"
Oct 30 19:47:30.390157 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:30.390115    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c\": container with ID starting with fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c not found: ID does not exist" containerID="fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c"
Oct 30 19:47:30.390297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.390154    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c"} err="failed to get container status \"fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c\": rpc error: code = NotFound desc = could not find container \"fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c\": container with ID starting with fc4c71d452b3f5d4bc8310da98cd1348d7b1f396fd01212be30af3dfe2baea1c not found: ID does not exist"
Oct 30 19:47:30.393996 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:30.393965    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-qkmqx"]
Oct 30 19:47:32.114251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:32.114215    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2467237f-3ab9-424a-9899-37f77850510a" path="/var/lib/kubelet/pods/2467237f-3ab9-424a-9899-37f77850510a/volumes"
Oct 30 19:47:34.324660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:34.324625    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:47:36.883461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:36.883427    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"]
Oct 30 19:47:36.883978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:36.883661    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf" podUID="ae2ceff9-2813-4c3d-8847-8ddbbab67904" containerName="maas-api" containerID="cri-o://659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4" gracePeriod=30
Oct 30 19:47:37.003073 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.003007    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"]
Oct 30 19:47:37.003403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.003378    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5" podUID="ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0" containerName="payload-pre-processing" containerID="cri-o://cb9c10636a896956bc174f9e399e052765a51fa2f2e248a28f9406632c64e8b4" gracePeriod=30
Oct 30 19:47:37.417295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.385294    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:47:37.487773 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.486003    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2ceff9-2813-4c3d-8847-8ddbbab67904-kube-api-access-4h5kj\" (UniqueName: \"kubernetes.io/projected/ae2ceff9-2813-4c3d-8847-8ddbbab67904-kube-api-access-4h5kj\") pod \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") "
Oct 30 19:47:37.488160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.487392    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-maas-api-tls\") pod \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") "
Oct 30 19:47:37.499979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.489985    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-metrics-certs\") pod \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") "
Oct 30 19:47:37.499979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.493573    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2ceff9-2813-4c3d-8847-8ddbbab67904-openshift-service-ca" pod "ae2ceff9-2813-4c3d-8847-8ddbbab67904" (UID: "ae2ceff9-2813-4c3d-8847-8ddbbab67904"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:37.499979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.493646    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2ceff9-2813-4c3d-8847-8ddbbab67904-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2ceff9-2813-4c3d-8847-8ddbbab67904-openshift-service-ca\") pod \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\" (UID: \"ae2ceff9-2813-4c3d-8847-8ddbbab67904\") "
Oct 30 19:47:37.499979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.497741    6777 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2ceff9-2813-4c3d-8847-8ddbbab67904-openshift-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.516077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.513012    6777 generic.go:356] "Generic (PLEG): container finished" podID="ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0" containerID="cb9c10636a896956bc174f9e399e052765a51fa2f2e248a28f9406632c64e8b4" exitCode=0
Oct 30 19:47:37.516077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.513285    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5" event={"ID":"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0","Type":"ContainerDied","Data":"cb9c10636a896956bc174f9e399e052765a51fa2f2e248a28f9406632c64e8b4"}
Oct 30 19:47:37.522236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.516514    6777 generic.go:356] "Generic (PLEG): container finished" podID="ae2ceff9-2813-4c3d-8847-8ddbbab67904" containerID="659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4" exitCode=0
Oct 30 19:47:37.522236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.516727    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf" event={"ID":"ae2ceff9-2813-4c3d-8847-8ddbbab67904","Type":"ContainerDied","Data":"659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4"}
Oct 30 19:47:37.522236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.516780    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf" event={"ID":"ae2ceff9-2813-4c3d-8847-8ddbbab67904","Type":"ContainerDied","Data":"8a981f770d2f5588a2ee6c501cb59e39ed5d52870f9ffb17adf46313909f610c"}
Oct 30 19:47:37.522236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.516813    6777 scope.go:122] "RemoveContainer" containerID="659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4"
Oct 30 19:47:37.522236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.516967    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"
Oct 30 19:47:37.522236 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.519618    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-metrics-certs" pod "ae2ceff9-2813-4c3d-8847-8ddbbab67904" (UID: "ae2ceff9-2813-4c3d-8847-8ddbbab67904"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:37.533089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.533002    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-maas-api-tls" pod "ae2ceff9-2813-4c3d-8847-8ddbbab67904" (UID: "ae2ceff9-2813-4c3d-8847-8ddbbab67904"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:37.540870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.540803    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2ceff9-2813-4c3d-8847-8ddbbab67904-kube-api-access-4h5kj" pod "ae2ceff9-2813-4c3d-8847-8ddbbab67904" (UID: "ae2ceff9-2813-4c3d-8847-8ddbbab67904"). InnerVolumeSpecName "kube-api-access-4h5kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:37.566502 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.566386    6777 scope.go:122] "RemoveContainer" containerID="659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4"
Oct 30 19:47:37.566746 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:37.566717    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4\": container with ID starting with 659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4 not found: ID does not exist" containerID="659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4"
Oct 30 19:47:37.566868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.566758    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4"} err="failed to get container status \"659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4\": rpc error: code = NotFound desc = could not find container \"659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4\": container with ID starting with 659fe1ed39d0a5eddc86fd2c85fe357f5b0763b76d7dff2a1a4a8278c3c844c4 not found: ID does not exist"
Oct 30 19:47:37.569166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.568727    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:47:37.633664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.633594    6777 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-metrics-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.633664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.633636    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5kj\" (UniqueName: \"kubernetes.io/projected/ae2ceff9-2813-4c3d-8847-8ddbbab67904-kube-api-access-4h5kj\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.633664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.633658    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2ceff9-2813-4c3d-8847-8ddbbab67904-maas-api-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.734272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.734232    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-plugins-config-volume\") pod \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\" (UID: \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\") "
Oct 30 19:47:37.734489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.734441    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-kube-api-access-7558c\" (UniqueName: \"kubernetes.io/projected/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-kube-api-access-7558c\") pod \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\" (UID: \"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0\") "
Oct 30 19:47:37.734658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.734632    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-plugins-config-volume" pod "ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0" (UID: "ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:37.735030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.735000    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.736597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.736575    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-kube-api-access-7558c" pod "ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0" (UID: "ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0"). InnerVolumeSpecName "kube-api-access-7558c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:37.843815 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.843779    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7558c\" (UniqueName: \"kubernetes.io/projected/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0-kube-api-access-7558c\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.859024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.858982    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"]
Oct 30 19:47:37.864289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:37.864259    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c2b092-d999899df-ln9hf"]
Oct 30 19:47:38.114400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:38.114319    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2ceff9-2813-4c3d-8847-8ddbbab67904" path="/var/lib/kubelet/pods/ae2ceff9-2813-4c3d-8847-8ddbbab67904/volumes"
Oct 30 19:47:38.323414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:38.323384    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:47:38.526701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:38.526609    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5" event={"ID":"ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0","Type":"ContainerDied","Data":"c8274e0296a9f24bbd12506a07696073573baee20acafb7230553889f56916a0"}
Oct 30 19:47:38.526701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:38.526644    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"
Oct 30 19:47:38.526701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:38.526677    6777 scope.go:122] "RemoveContainer" containerID="cb9c10636a896956bc174f9e399e052765a51fa2f2e248a28f9406632c64e8b4"
Oct 30 19:47:38.549489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:38.549455    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"]
Oct 30 19:47:38.553284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:38.553254    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2b092-5757dbb7f9-zf2r5"]
Oct 30 19:47:40.114220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:40.114186    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0" path="/var/lib/kubelet/pods/ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0/volumes"
Oct 30 19:47:40.907983 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:40.907943    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"]
Oct 30 19:47:40.908264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:40.908238    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy" containerID="cri-o://b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb" gracePeriod=30
Oct 30 19:47:40.921755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:40.921720    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"]
Oct 30 19:47:40.921955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:40.921935    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy" containerID="cri-o://ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525" gracePeriod=30
Oct 30 19:47:42.603151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:42.603124    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 19:47:42.903671 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:42.903591    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 19:47:43.641583 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:43.641545    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:47:43.728924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.728887    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"]
Oct 30 19:47:43.729898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.729867    6777 cpu_manager.go:414] "containerMap: removing container" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy"
Oct 30 19:47:43.729898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.729893    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.729916    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0" containerName="payload-pre-processing"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.729925    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0" containerName="payload-pre-processing"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.729956    6777 cpu_manager.go:414] "containerMap: removing container" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.729963    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730001    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ae2ceff9-2813-4c3d-8847-8ddbbab67904" containerName="maas-api"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730009    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2ceff9-2813-4c3d-8847-8ddbbab67904" containerName="maas-api"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730022    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730027    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730050    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy"
Oct 30 19:47:43.730147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730074    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7327b5-03ab-49fa-89e7-08df93bf233d" containerName="istio-proxy"
Oct 30 19:47:43.730626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730223    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy"
Oct 30 19:47:43.730626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730239    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae58dd48-af4d-4fc9-b0ce-3ef47507e4b0" containerName="payload-pre-processing"
Oct 30 19:47:43.730626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730254    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="2467237f-3ab9-424a-9899-37f77850510a" containerName="istio-proxy"
Oct 30 19:47:43.730626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730261    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfd0a26-981e-4313-b6d4-66dbf0107982" containerName="istio-proxy"
Oct 30 19:47:43.730626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.730276    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2ceff9-2813-4c3d-8847-8ddbbab67904" containerName="maas-api"
Oct 30 19:47:43.735677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.735656    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:43.740732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.740705    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:43.741134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.741111    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"]
Oct 30 19:47:43.741264 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.741246    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:47:43.896705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.896615    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91275d29-3636-4502-9bff-c624dceebc77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:43.896903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.896780    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91275d29-3636-4502-9bff-c624dceebc77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:43.897004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.896982    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48pwx\" (UniqueName: \"kubernetes.io/projected/91275d29-3636-4502-9bff-c624dceebc77-kube-api-access-48pwx\") pod \"maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:43.998805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.998766    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91275d29-3636-4502-9bff-c624dceebc77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:43.998976 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.998924    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48pwx\" (UniqueName: \"kubernetes.io/projected/91275d29-3636-4502-9bff-c624dceebc77-kube-api-access-48pwx\") pod \"maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:43.999112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.998974    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91275d29-3636-4502-9bff-c624dceebc77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:43.999565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:43.999539    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91275d29-3636-4502-9bff-c624dceebc77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:44.001525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:44.001504    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91275d29-3636-4502-9bff-c624dceebc77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:44.012500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:44.012465    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48pwx\" (UniqueName: \"kubernetes.io/projected/91275d29-3636-4502-9bff-c624dceebc77-kube-api-access-48pwx\") pod \"maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:44.053716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:44.053666    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:44.335275 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:44.335246    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"]
Oct 30 19:47:44.336582 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:47:44.336556    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91275d29_3636_4502_9bff_c624dceebc77.slice/crio-f013fd58da72d647360155b298bb57ab5206476aa2138be6c691a07f07b717ab WatchSource:0}: Error finding container f013fd58da72d647360155b298bb57ab5206476aa2138be6c691a07f07b717ab: Status 404 returned error can't find the container with id f013fd58da72d647360155b298bb57ab5206476aa2138be6c691a07f07b717ab
Oct 30 19:47:44.592848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:44.592752    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz" event={"ID":"91275d29-3636-4502-9bff-c624dceebc77","Type":"ContainerStarted","Data":"f013fd58da72d647360155b298bb57ab5206476aa2138be6c691a07f07b717ab"}
Oct 30 19:47:45.601487 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:45.601453    6777 generic.go:356] "Generic (PLEG): container finished" podID="91275d29-3636-4502-9bff-c624dceebc77" containerID="09d4be9b5a684e47842b72ccc623f2ebc7d111f824fc9f42e4f7e061afec5c0a" exitCode=0
Oct 30 19:47:45.601930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:45.601540    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz" event={"ID":"91275d29-3636-4502-9bff-c624dceebc77","Type":"ContainerDied","Data":"09d4be9b5a684e47842b72ccc623f2ebc7d111f824fc9f42e4f7e061afec5c0a"}
Oct 30 19:47:46.238795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.238752    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"]
Oct 30 19:47:46.239404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.239371    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy" containerID="cri-o://0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b" gracePeriod=30
Oct 30 19:47:46.247641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.247588    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"]
Oct 30 19:47:46.248052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.247997    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy" containerID="cri-o://8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af" gracePeriod=30
Oct 30 19:47:46.276423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.274745    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:47:46.455402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.453910    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-socket\") pod \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") "
Oct 30 19:47:46.455402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.453968    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-certs\") pod \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") "
Oct 30 19:47:46.455402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.454040    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istiod-ca-cert\") pod \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") "
Oct 30 19:47:46.455402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.454092    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-kube-api-access-wv8m6\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-kube-api-access-wv8m6\") pod \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") "
Oct 30 19:47:46.455402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.454195    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-token\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-token\") pod \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") "
Oct 30 19:47:46.455402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.454909    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-certs" pod "ba8d5446-5b96-41f1-815e-88c0d09b24a0" (UID: "ba8d5446-5b96-41f1-815e-88c0d09b24a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.455402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.455183    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-socket" pod "ba8d5446-5b96-41f1-815e-88c0d09b24a0" (UID: "ba8d5446-5b96-41f1-815e-88c0d09b24a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.456218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.455469    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istiod-ca-cert" pod "ba8d5446-5b96-41f1-815e-88c0d09b24a0" (UID: "ba8d5446-5b96-41f1-815e-88c0d09b24a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:46.459392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.458495    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.459392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.458526    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.459392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.458539    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.483458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.483421    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-kube-api-access-wv8m6" pod "ba8d5446-5b96-41f1-815e-88c0d09b24a0" (UID: "ba8d5446-5b96-41f1-815e-88c0d09b24a0"). InnerVolumeSpecName "kube-api-access-wv8m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.485445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.485408    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-token" pod "ba8d5446-5b96-41f1-815e-88c0d09b24a0" (UID: "ba8d5446-5b96-41f1-815e-88c0d09b24a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.520569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.520544    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:47:46.559888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.559851    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-data\") pod \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") "
Oct 30 19:47:46.560392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.560361    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-podinfo\") pod \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") "
Oct 30 19:47:46.560616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.560598    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-credential-socket\") pod \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") "
Oct 30 19:47:46.560941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.560924    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-envoy\") pod \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\" (UID: \"ba8d5446-5b96-41f1-815e-88c0d09b24a0\") "
Oct 30 19:47:46.562396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.562373    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.562561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.562545    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv8m6\" (UniqueName: \"kubernetes.io/projected/ba8d5446-5b96-41f1-815e-88c0d09b24a0-kube-api-access-wv8m6\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.562670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.561083    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-credential-socket" pod "ba8d5446-5b96-41f1-815e-88c0d09b24a0" (UID: "ba8d5446-5b96-41f1-815e-88c0d09b24a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.563636 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.563608    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-envoy" pod "ba8d5446-5b96-41f1-815e-88c0d09b24a0" (UID: "ba8d5446-5b96-41f1-815e-88c0d09b24a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.563764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.563745    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-data" pod "ba8d5446-5b96-41f1-815e-88c0d09b24a0" (UID: "ba8d5446-5b96-41f1-815e-88c0d09b24a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.571215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.571180    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-podinfo" pod "ba8d5446-5b96-41f1-815e-88c0d09b24a0" (UID: "ba8d5446-5b96-41f1-815e-88c0d09b24a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:46.613641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.613606    6777 generic.go:356] "Generic (PLEG): container finished" podID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerID="b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb" exitCode=0
Oct 30 19:47:46.614182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.613679    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"
Oct 30 19:47:46.614182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.613685    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" event={"ID":"ba8d5446-5b96-41f1-815e-88c0d09b24a0","Type":"ContainerDied","Data":"b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb"}
Oct 30 19:47:46.614182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.613768    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc" event={"ID":"ba8d5446-5b96-41f1-815e-88c0d09b24a0","Type":"ContainerDied","Data":"f87db58e4f032fcc85d7b396ee8732689d7e06d5012dd7ea85797a35b36b0ff4"}
Oct 30 19:47:46.614182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.613820    6777 scope.go:122] "RemoveContainer" containerID="b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb"
Oct 30 19:47:46.615709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.615672    6777 generic.go:356] "Generic (PLEG): container finished" podID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerID="ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525" exitCode=0
Oct 30 19:47:46.616265 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.615779    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"
Oct 30 19:47:46.616617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.615795    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" event={"ID":"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4","Type":"ContainerDied","Data":"ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525"}
Oct 30 19:47:46.616617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.616485    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j" event={"ID":"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4","Type":"ContainerDied","Data":"6dfb0449c79f5552b0ddbb65609e128aafd9d38400407d36902fbbc01f87892e"}
Oct 30 19:47:46.633956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.633931    6777 scope.go:122] "RemoveContainer" containerID="0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5"
Oct 30 19:47:46.653591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.653510    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"]
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.662228    6777 scope.go:122] "RemoveContainer" containerID="b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb"
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:46.662876    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb\": container with ID starting with b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb not found: ID does not exist" containerID="b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb"
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.662914    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb"} err="failed to get container status \"b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb\": rpc error: code = NotFound desc = could not find container \"b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb\": container with ID starting with b8942b35aafb7c87524978085fd856d541d85802c8678c4c39134a59bea13afb not found: ID does not exist"
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.662941    6777 scope.go:122] "RemoveContainer" containerID="0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5"
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.663505    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-token\") pod \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") "
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.663570    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-data\") pod \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") "
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.663640    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-kube-api-access-h5tj5\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-kube-api-access-h5tj5\") pod \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") "
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.663686    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istiod-ca-cert\") pod \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") "
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.663727    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-credential-socket\") pod \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") "
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.663797    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-certs\") pod \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") "
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.663883    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-socket\") pod \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") "
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.663918    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-envoy\") pod \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") "
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.664092    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-podinfo\") pod \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\" (UID: \"9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4\") "
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.664750    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.664771    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.664785    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8d5446-5b96-41f1-815e-88c0d09b24a0-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.665210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.664800    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8d5446-5b96-41f1-815e-88c0d09b24a0-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.668764 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:46.665757    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5\": container with ID starting with 0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5 not found: ID does not exist" containerID="0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5"
Oct 30 19:47:46.668764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.665795    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5"} err="failed to get container status \"0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5\": rpc error: code = NotFound desc = could not find container \"0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5\": container with ID starting with 0b9c8c2787a1c854286b58de4a42f5971547a8eb5fe8aa826b9ba3936d0216b5 not found: ID does not exist"
Oct 30 19:47:46.668764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.665820    6777 scope.go:122] "RemoveContainer" containerID="ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525"
Oct 30 19:47:46.668764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.668023    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-zfzrc"]
Oct 30 19:47:46.670179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.669275    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-socket" pod "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" (UID: "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.670179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.669533    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-certs" pod "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" (UID: "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.670179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.669751    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-data" pod "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" (UID: "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.670179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.669863    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istiod-ca-cert" pod "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" (UID: "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:46.670179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.669993    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-credential-socket" pod "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" (UID: "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.672291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.672204    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-envoy" pod "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" (UID: "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.672690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.672654    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-token" pod "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" (UID: "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.673674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.673641    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-kube-api-access-h5tj5" pod "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" (UID: "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4"). InnerVolumeSpecName "kube-api-access-h5tj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.680274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.680218    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-podinfo" pod "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" (UID: "9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:46.690457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.688932    6777 scope.go:122] "RemoveContainer" containerID="ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525"
Oct 30 19:47:46.691003 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:46.690973    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525\": container with ID starting with ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525 not found: ID does not exist" containerID="ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525"
Oct 30 19:47:46.691234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.691008    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525"} err="failed to get container status \"ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525\": rpc error: code = NotFound desc = could not find container \"ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525\": container with ID starting with ef7bd430a9fa25d7a656275d766fb97ffca6c3af69e2339f73e9f5a9edf8b525 not found: ID does not exist"
Oct 30 19:47:46.766304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.766200    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.766304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.766234    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.766304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.766251    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tj5\" (UniqueName: \"kubernetes.io/projected/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-kube-api-access-h5tj5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.766304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.766265    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.766304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.766279    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.766304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.766293    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.766304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.766307    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.766830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.766320    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.766830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.766332    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.801457 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.801422    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:46.867543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.867500    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91275d29-3636-4502-9bff-c624dceebc77-kube-api-access-48pwx\" (UniqueName: \"kubernetes.io/projected/91275d29-3636-4502-9bff-c624dceebc77-kube-api-access-48pwx\") pod \"91275d29-3636-4502-9bff-c624dceebc77\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") "
Oct 30 19:47:46.867762 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.867717    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91275d29-3636-4502-9bff-c624dceebc77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91275d29-3636-4502-9bff-c624dceebc77-maas-api-openshift-ca\") pod \"91275d29-3636-4502-9bff-c624dceebc77\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") "
Oct 30 19:47:46.868198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.867880    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91275d29-3636-4502-9bff-c624dceebc77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91275d29-3636-4502-9bff-c624dceebc77-maas-api-xks-ca\") pod \"91275d29-3636-4502-9bff-c624dceebc77\" (UID: \"91275d29-3636-4502-9bff-c624dceebc77\") "
Oct 30 19:47:46.868198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.868153    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91275d29-3636-4502-9bff-c624dceebc77-maas-api-openshift-ca" pod "91275d29-3636-4502-9bff-c624dceebc77" (UID: "91275d29-3636-4502-9bff-c624dceebc77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:46.868688 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.868645    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91275d29-3636-4502-9bff-c624dceebc77-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.870003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.869974    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91275d29-3636-4502-9bff-c624dceebc77-kube-api-access-48pwx" pod "91275d29-3636-4502-9bff-c624dceebc77" (UID: "91275d29-3636-4502-9bff-c624dceebc77"). InnerVolumeSpecName "kube-api-access-48pwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.870583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.870556    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91275d29-3636-4502-9bff-c624dceebc77-maas-api-xks-ca" pod "91275d29-3636-4502-9bff-c624dceebc77" (UID: "91275d29-3636-4502-9bff-c624dceebc77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:46.943982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.943678    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"]
Oct 30 19:47:46.947190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.947161    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-jf58j"]
Oct 30 19:47:46.970086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.970039    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48pwx\" (UniqueName: \"kubernetes.io/projected/91275d29-3636-4502-9bff-c624dceebc77-kube-api-access-48pwx\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.970086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:46.970083    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91275d29-3636-4502-9bff-c624dceebc77-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:47.626306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:47.626276    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"
Oct 30 19:47:47.626306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:47.626293    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz" event={"ID":"91275d29-3636-4502-9bff-c624dceebc77","Type":"ContainerDied","Data":"f013fd58da72d647360155b298bb57ab5206476aa2138be6c691a07f07b717ab"}
Oct 30 19:47:47.626877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:47.626347    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f013fd58da72d647360155b298bb57ab5206476aa2138be6c691a07f07b717ab"
Oct 30 19:47:48.114298 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:48.114261    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" path="/var/lib/kubelet/pods/9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4/volumes"
Oct 30 19:47:48.114816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:48.114798    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" path="/var/lib/kubelet/pods/ba8d5446-5b96-41f1-815e-88c0d09b24a0/volumes"
Oct 30 19:47:50.314171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.314123    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"]
Oct 30 19:47:50.314704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.314370    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd" podUID="cdffbd8e-5209-4b25-b923-92e782647e4b" containerName="payload-processing" containerID="cri-o://7d3f0613118d3827868551e20f6000ba2b73d82a9e483b7bd884231bd3afe015" gracePeriod=30
Oct 30 19:47:50.325294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.325264    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"]
Oct 30 19:47:50.325531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.325509    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79" podUID="4a4dd36a-3268-45c8-a86e-a37c70e77081" containerName="payload-pre-processing" containerID="cri-o://6f965a753b6c97692bb4cf10a5eb3b803edda85d281bc3abb103e9df3aa642dd" gracePeriod=30
Oct 30 19:47:50.672116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.671743    6777 generic.go:356] "Generic (PLEG): container finished" podID="4a4dd36a-3268-45c8-a86e-a37c70e77081" containerID="6f965a753b6c97692bb4cf10a5eb3b803edda85d281bc3abb103e9df3aa642dd" exitCode=0
Oct 30 19:47:50.672116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.671885    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79" event={"ID":"4a4dd36a-3268-45c8-a86e-a37c70e77081","Type":"ContainerDied","Data":"6f965a753b6c97692bb4cf10a5eb3b803edda85d281bc3abb103e9df3aa642dd"}
Oct 30 19:47:50.679482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.679390    6777 generic.go:356] "Generic (PLEG): container finished" podID="cdffbd8e-5209-4b25-b923-92e782647e4b" containerID="7d3f0613118d3827868551e20f6000ba2b73d82a9e483b7bd884231bd3afe015" exitCode=0
Oct 30 19:47:50.679684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.679546    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd" event={"ID":"cdffbd8e-5209-4b25-b923-92e782647e4b","Type":"ContainerDied","Data":"7d3f0613118d3827868551e20f6000ba2b73d82a9e483b7bd884231bd3afe015"}
Oct 30 19:47:50.776904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.776872    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:47:50.817835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.817804    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:47:50.973482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.973376    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdffbd8e-5209-4b25-b923-92e782647e4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdffbd8e-5209-4b25-b923-92e782647e4b-plugins-config-volume\") pod \"cdffbd8e-5209-4b25-b923-92e782647e4b\" (UID: \"cdffbd8e-5209-4b25-b923-92e782647e4b\") "
Oct 30 19:47:50.973693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.973535    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4dd36a-3268-45c8-a86e-a37c70e77081-kube-api-access-l4rzn\" (UniqueName: \"kubernetes.io/projected/4a4dd36a-3268-45c8-a86e-a37c70e77081-kube-api-access-l4rzn\") pod \"4a4dd36a-3268-45c8-a86e-a37c70e77081\" (UID: \"4a4dd36a-3268-45c8-a86e-a37c70e77081\") "
Oct 30 19:47:50.973693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.973633    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a4dd36a-3268-45c8-a86e-a37c70e77081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4dd36a-3268-45c8-a86e-a37c70e77081-plugins-config-volume\") pod \"4a4dd36a-3268-45c8-a86e-a37c70e77081\" (UID: \"4a4dd36a-3268-45c8-a86e-a37c70e77081\") "
Oct 30 19:47:50.977972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.974846    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdffbd8e-5209-4b25-b923-92e782647e4b-kube-api-access-fgqcm\" (UniqueName: \"kubernetes.io/projected/cdffbd8e-5209-4b25-b923-92e782647e4b-kube-api-access-fgqcm\") pod \"cdffbd8e-5209-4b25-b923-92e782647e4b\" (UID: \"cdffbd8e-5209-4b25-b923-92e782647e4b\") "
Oct 30 19:47:50.977972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.976772    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdffbd8e-5209-4b25-b923-92e782647e4b-plugins-config-volume" pod "cdffbd8e-5209-4b25-b923-92e782647e4b" (UID: "cdffbd8e-5209-4b25-b923-92e782647e4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:50.979230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.979191    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4dd36a-3268-45c8-a86e-a37c70e77081-kube-api-access-l4rzn" pod "4a4dd36a-3268-45c8-a86e-a37c70e77081" (UID: "4a4dd36a-3268-45c8-a86e-a37c70e77081"). InnerVolumeSpecName "kube-api-access-l4rzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:50.980208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.980182    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4dd36a-3268-45c8-a86e-a37c70e77081-plugins-config-volume" pod "4a4dd36a-3268-45c8-a86e-a37c70e77081" (UID: "4a4dd36a-3268-45c8-a86e-a37c70e77081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:50.982477 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:50.982448    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdffbd8e-5209-4b25-b923-92e782647e4b-kube-api-access-fgqcm" pod "cdffbd8e-5209-4b25-b923-92e782647e4b" (UID: "cdffbd8e-5209-4b25-b923-92e782647e4b"). InnerVolumeSpecName "kube-api-access-fgqcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.087387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.085967    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdffbd8e-5209-4b25-b923-92e782647e4b-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.087387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.086012    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rzn\" (UniqueName: \"kubernetes.io/projected/4a4dd36a-3268-45c8-a86e-a37c70e77081-kube-api-access-l4rzn\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.087387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.086027    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4dd36a-3268-45c8-a86e-a37c70e77081-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.087387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.086040    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgqcm\" (UniqueName: \"kubernetes.io/projected/cdffbd8e-5209-4b25-b923-92e782647e4b-kube-api-access-fgqcm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.488609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.488576    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:47:51.568155 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.568114    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"]
Oct 30 19:47:51.586777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.586741    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"]
Oct 30 19:47:51.595620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.595584    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-credential-socket\") pod \"86401767-2d4b-430d-bfd1-11208842daca\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") "
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.595687    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-data\") pod \"86401767-2d4b-430d-bfd1-11208842daca\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") "
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.595748    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-certs\") pod \"86401767-2d4b-430d-bfd1-11208842daca\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") "
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.595854    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-socket\") pod \"86401767-2d4b-430d-bfd1-11208842daca\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") "
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.595883    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-kube-api-access-bfnvv\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-kube-api-access-bfnvv\") pod \"86401767-2d4b-430d-bfd1-11208842daca\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") "
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.595921    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-istio-token\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-istio-token\") pod \"86401767-2d4b-430d-bfd1-11208842daca\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") "
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.595951    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86401767-2d4b-430d-bfd1-11208842daca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86401767-2d4b-430d-bfd1-11208842daca-istiod-ca-cert\") pod \"86401767-2d4b-430d-bfd1-11208842daca\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") "
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.596074    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86401767-2d4b-430d-bfd1-11208842daca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86401767-2d4b-430d-bfd1-11208842daca-istio-podinfo\") pod \"86401767-2d4b-430d-bfd1-11208842daca\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") "
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.596130    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-envoy\") pod \"86401767-2d4b-430d-bfd1-11208842daca\" (UID: \"86401767-2d4b-430d-bfd1-11208842daca\") "
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.596135    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-credential-socket" pod "86401767-2d4b-430d-bfd1-11208842daca" (UID: "86401767-2d4b-430d-bfd1-11208842daca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.596232    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-socket" pod "86401767-2d4b-430d-bfd1-11208842daca" (UID: "86401767-2d4b-430d-bfd1-11208842daca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.596412    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-data" pod "86401767-2d4b-430d-bfd1-11208842daca" (UID: "86401767-2d4b-430d-bfd1-11208842daca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.596826    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.596844    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.596919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.596861    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.597727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.597082    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-certs" pod "86401767-2d4b-430d-bfd1-11208842daca" (UID: "86401767-2d4b-430d-bfd1-11208842daca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.597727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.597365    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86401767-2d4b-430d-bfd1-11208842daca-istiod-ca-cert" pod "86401767-2d4b-430d-bfd1-11208842daca" (UID: "86401767-2d4b-430d-bfd1-11208842daca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:51.599273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.599198    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-envoy" pod "86401767-2d4b-430d-bfd1-11208842daca" (UID: "86401767-2d4b-430d-bfd1-11208842daca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.600462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.600418    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86401767-2d4b-430d-bfd1-11208842daca-istio-podinfo" pod "86401767-2d4b-430d-bfd1-11208842daca" (UID: "86401767-2d4b-430d-bfd1-11208842daca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:51.602508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.601762    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-istio-token" pod "86401767-2d4b-430d-bfd1-11208842daca" (UID: "86401767-2d4b-430d-bfd1-11208842daca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.602508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.601924    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-kube-api-access-bfnvv" pod "86401767-2d4b-430d-bfd1-11208842daca" (UID: "86401767-2d4b-430d-bfd1-11208842daca"). InnerVolumeSpecName "kube-api-access-bfnvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.605441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.605419    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:47:51.694435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.694385    6777 generic.go:356] "Generic (PLEG): container finished" podID="86401767-2d4b-430d-bfd1-11208842daca" containerID="0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b" exitCode=0
Oct 30 19:47:51.695872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.695851    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"
Oct 30 19:47:51.701409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.701085    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" event={"ID":"86401767-2d4b-430d-bfd1-11208842daca","Type":"ContainerDied","Data":"0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b"}
Oct 30 19:47:51.701409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.701157    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d" event={"ID":"86401767-2d4b-430d-bfd1-11208842daca","Type":"ContainerDied","Data":"5f7e98e112f6b57c984a2e4e4c3002a2c0c8b5816eeddfa810dbabaca0efc10f"}
Oct 30 19:47:51.701409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.701185    6777 scope.go:122] "RemoveContainer" containerID="0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b"
Oct 30 19:47:51.701729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.701694    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-certs\") pod \"85923a0f-3daa-4295-9a1d-3ca15689f158\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") "
Oct 30 19:47:51.701817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.701747    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-kube-api-access-8grbw\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-kube-api-access-8grbw\") pod \"85923a0f-3daa-4295-9a1d-3ca15689f158\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") "
Oct 30 19:47:51.701817 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.701796    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-token\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-token\") pod \"85923a0f-3daa-4295-9a1d-3ca15689f158\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") "
Oct 30 19:47:51.701960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.701934    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-certs" pod "85923a0f-3daa-4295-9a1d-3ca15689f158" (UID: "85923a0f-3daa-4295-9a1d-3ca15689f158"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.702442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702414    6777 generic.go:356] "Generic (PLEG): container finished" podID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerID="8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af" exitCode=0
Oct 30 19:47:51.702748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702461    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86401767-2d4b-430d-bfd1-11208842daca-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.702860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702759    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86401767-2d4b-430d-bfd1-11208842daca-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.702860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702776    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.702860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702507    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" event={"ID":"85923a0f-3daa-4295-9a1d-3ca15689f158","Type":"ContainerDied","Data":"8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af"}
Oct 30 19:47:51.702860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702790    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.702860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702804    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86401767-2d4b-430d-bfd1-11208842daca-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.702860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702818    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfnvv\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-kube-api-access-bfnvv\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.702860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702832    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86401767-2d4b-430d-bfd1-11208842daca-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.702860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702830    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r" event={"ID":"85923a0f-3daa-4295-9a1d-3ca15689f158","Type":"ContainerDied","Data":"48133218ca05a5b211555ebfef19bfa8f9a13c5fe7fb5ec4bae69e8130d2cfa1"}
Oct 30 19:47:51.703408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.702528    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"
Oct 30 19:47:51.707120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.704503    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"
Oct 30 19:47:51.707120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.704608    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79" event={"ID":"4a4dd36a-3268-45c8-a86e-a37c70e77081","Type":"ContainerDied","Data":"45ce7f7f63ead5032d6d26dbaf6e3db05e70eb254042fd854ca346e62f2af70b"}
Oct 30 19:47:51.707120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.705190    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-kube-api-access-8grbw" pod "85923a0f-3daa-4295-9a1d-3ca15689f158" (UID: "85923a0f-3daa-4295-9a1d-3ca15689f158"). InnerVolumeSpecName "kube-api-access-8grbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.708332 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.708307    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd" event={"ID":"cdffbd8e-5209-4b25-b923-92e782647e4b","Type":"ContainerDied","Data":"11780c8385eede051843dcc3bdf4cd6e52b1fff54c0a44f94be355f01aa999eb"}
Oct 30 19:47:51.708449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.708345    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"
Oct 30 19:47:51.709648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.709623    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-token" pod "85923a0f-3daa-4295-9a1d-3ca15689f158" (UID: "85923a0f-3daa-4295-9a1d-3ca15689f158"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.716993 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.716971    6777 scope.go:122] "RemoveContainer" containerID="0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b"
Oct 30 19:47:51.717347 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:51.717320    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b\": container with ID starting with 0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b not found: ID does not exist" containerID="0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b"
Oct 30 19:47:51.717443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.717357    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b"} err="failed to get container status \"0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b\": rpc error: code = NotFound desc = could not find container \"0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b\": container with ID starting with 0f8131307cf14c4651bcd7d54c95440f2c78f28c6f4b63fc8feb60418a65652b not found: ID does not exist"
Oct 30 19:47:51.717443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.717376    6777 scope.go:122] "RemoveContainer" containerID="8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af"
Oct 30 19:47:51.728547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.728525    6777 scope.go:122] "RemoveContainer" containerID="94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549"
Oct 30 19:47:51.739209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.739186    6777 scope.go:122] "RemoveContainer" containerID="8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af"
Oct 30 19:47:51.739574 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:51.739542    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af\": container with ID starting with 8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af not found: ID does not exist" containerID="8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af"
Oct 30 19:47:51.739681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.739585    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af"} err="failed to get container status \"8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af\": rpc error: code = NotFound desc = could not find container \"8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af\": container with ID starting with 8700cc0beeda4fecf02d05722809d6b4829743172139803fd729c835547f48af not found: ID does not exist"
Oct 30 19:47:51.739681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.739609    6777 scope.go:122] "RemoveContainer" containerID="94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549"
Oct 30 19:47:51.739888 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:51.739866    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549\": container with ID starting with 94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549 not found: ID does not exist" containerID="94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549"
Oct 30 19:47:51.739941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.739895    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549"} err="failed to get container status \"94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549\": rpc error: code = NotFound desc = could not find container \"94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549\": container with ID starting with 94c13c0cbc6107cdfd8d5835f12d31945cf5b500eaee87f71db17151fdb1a549 not found: ID does not exist"
Oct 30 19:47:51.739941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.739911    6777 scope.go:122] "RemoveContainer" containerID="6f965a753b6c97692bb4cf10a5eb3b803edda85d281bc3abb103e9df3aa642dd"
Oct 30 19:47:51.747824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.747794    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"]
Oct 30 19:47:51.754153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.754121    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gbw8d"]
Oct 30 19:47:51.756936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.756921    6777 scope.go:122] "RemoveContainer" containerID="7d3f0613118d3827868551e20f6000ba2b73d82a9e483b7bd884231bd3afe015"
Oct 30 19:47:51.768745 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.768719    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"]
Oct 30 19:47:51.773378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.773350    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa49c5-69dfbdb5c9-vcb79"]
Oct 30 19:47:51.786225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.786195    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"]
Oct 30 19:47:51.790821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.790794    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa49c5-c48688486-ljmsd"]
Oct 30 19:47:51.803341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803254    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-socket\") pod \"85923a0f-3daa-4295-9a1d-3ca15689f158\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") "
Oct 30 19:47:51.803341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803295    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85923a0f-3daa-4295-9a1d-3ca15689f158-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85923a0f-3daa-4295-9a1d-3ca15689f158-istiod-ca-cert\") pod \"85923a0f-3daa-4295-9a1d-3ca15689f158\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") "
Oct 30 19:47:51.803341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803335    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-podinfo\") pod \"85923a0f-3daa-4295-9a1d-3ca15689f158\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") "
Oct 30 19:47:51.803601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803358    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-credential-socket\") pod \"85923a0f-3daa-4295-9a1d-3ca15689f158\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") "
Oct 30 19:47:51.803601 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803386    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-envoy\") pod \"85923a0f-3daa-4295-9a1d-3ca15689f158\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") "
Oct 30 19:47:51.803715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803594    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-socket" pod "85923a0f-3daa-4295-9a1d-3ca15689f158" (UID: "85923a0f-3daa-4295-9a1d-3ca15689f158"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.803715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803656    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-data\") pod \"85923a0f-3daa-4295-9a1d-3ca15689f158\" (UID: \"85923a0f-3daa-4295-9a1d-3ca15689f158\") "
Oct 30 19:47:51.803819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803657    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-credential-socket" pod "85923a0f-3daa-4295-9a1d-3ca15689f158" (UID: "85923a0f-3daa-4295-9a1d-3ca15689f158"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.803819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803728    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85923a0f-3daa-4295-9a1d-3ca15689f158-istiod-ca-cert" pod "85923a0f-3daa-4295-9a1d-3ca15689f158" (UID: "85923a0f-3daa-4295-9a1d-3ca15689f158"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:51.803898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.803812    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-data" pod "85923a0f-3daa-4295-9a1d-3ca15689f158" (UID: "85923a0f-3daa-4295-9a1d-3ca15689f158"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.804157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.804140    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.804230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.804160    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8grbw\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-kube-api-access-8grbw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.804230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.804170    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.804230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.804178    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.804230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.804187    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.804230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.804200    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85923a0f-3daa-4295-9a1d-3ca15689f158-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.805731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.805714    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-envoy" pod "85923a0f-3daa-4295-9a1d-3ca15689f158" (UID: "85923a0f-3daa-4295-9a1d-3ca15689f158"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.806123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.806104    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-podinfo" pod "85923a0f-3daa-4295-9a1d-3ca15689f158" (UID: "85923a0f-3daa-4295-9a1d-3ca15689f158"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:51.910036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.909994    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.910672 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:51.910638    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85923a0f-3daa-4295-9a1d-3ca15689f158-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:52.033997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:52.033952    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"]
Oct 30 19:47:52.037970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:52.037934    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-gj29r"]
Oct 30 19:47:52.118742 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:52.118626    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4dd36a-3268-45c8-a86e-a37c70e77081" path="/var/lib/kubelet/pods/4a4dd36a-3268-45c8-a86e-a37c70e77081/volumes"
Oct 30 19:47:52.119391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:52.119346    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" path="/var/lib/kubelet/pods/85923a0f-3daa-4295-9a1d-3ca15689f158/volumes"
Oct 30 19:47:52.120362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:52.120339    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86401767-2d4b-430d-bfd1-11208842daca" path="/var/lib/kubelet/pods/86401767-2d4b-430d-bfd1-11208842daca/volumes"
Oct 30 19:47:52.121863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:52.121842    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdffbd8e-5209-4b25-b923-92e782647e4b" path="/var/lib/kubelet/pods/cdffbd8e-5209-4b25-b923-92e782647e4b/volumes"
Oct 30 19:47:56.568701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.568647    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89" podUID="63b33f32-ffbe-4bbc-b34e-a439325206ab" containerName="payload-processing" containerID="cri-o://5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6" gracePeriod=25
Oct 30 19:47:56.587599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.587523    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5" podUID="e8401cdc-e5c5-4c7a-8f24-078287a136a7" containerName="payload-pre-processing" containerID="cri-o://460a65234e747931ccc758234e67dfde453038fd2728104d097d8e5bea0ec2e9" gracePeriod=25
Oct 30 19:47:56.780659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.780625    6777 generic.go:356] "Generic (PLEG): container finished" podID="e8401cdc-e5c5-4c7a-8f24-078287a136a7" containerID="460a65234e747931ccc758234e67dfde453038fd2728104d097d8e5bea0ec2e9" exitCode=0
Oct 30 19:47:56.780881 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.780697    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5" event={"ID":"e8401cdc-e5c5-4c7a-8f24-078287a136a7","Type":"ContainerDied","Data":"460a65234e747931ccc758234e67dfde453038fd2728104d097d8e5bea0ec2e9"}
Oct 30 19:47:56.888640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.888614    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:47:56.983486 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.983438    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8401cdc-e5c5-4c7a-8f24-078287a136a7-kube-api-access-4pdr2\" (UniqueName: \"kubernetes.io/projected/e8401cdc-e5c5-4c7a-8f24-078287a136a7-kube-api-access-4pdr2\") pod \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\" (UID: \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\") "
Oct 30 19:47:56.983686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.983586    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8401cdc-e5c5-4c7a-8f24-078287a136a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8401cdc-e5c5-4c7a-8f24-078287a136a7-plugins-config-volume\") pod \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\" (UID: \"e8401cdc-e5c5-4c7a-8f24-078287a136a7\") "
Oct 30 19:47:56.983953 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.983924    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8401cdc-e5c5-4c7a-8f24-078287a136a7-plugins-config-volume" pod "e8401cdc-e5c5-4c7a-8f24-078287a136a7" (UID: "e8401cdc-e5c5-4c7a-8f24-078287a136a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:56.984230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.984216    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8401cdc-e5c5-4c7a-8f24-078287a136a7-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:56.985661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:56.985638    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8401cdc-e5c5-4c7a-8f24-078287a136a7-kube-api-access-4pdr2" pod "e8401cdc-e5c5-4c7a-8f24-078287a136a7" (UID: "e8401cdc-e5c5-4c7a-8f24-078287a136a7"). InnerVolumeSpecName "kube-api-access-4pdr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:57.085522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.085487    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pdr2\" (UniqueName: \"kubernetes.io/projected/e8401cdc-e5c5-4c7a-8f24-078287a136a7-kube-api-access-4pdr2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:57.378403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.378335    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"]
Oct 30 19:47:57.379354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.379321    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk" podUID="43f115ae-b735-46cd-a150-2c525e2f325d" containerName="payload-processing" containerID="cri-o://18080aa1aebcbd462bd5a3d7fc1997223b328717f3c0072c884bf7a9dfd1a122" gracePeriod=30
Oct 30 19:47:57.400311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.400276    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"]
Oct 30 19:47:57.400541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.400516    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2" podUID="0b305198-770f-4d2b-8f37-edccc157cddf" containerName="payload-pre-processing" containerID="cri-o://3b24869fd31e9a0e2cbc8cd9ae6884b08497350f5fa6979b3175053b52f8ca92" gracePeriod=30
Oct 30 19:47:57.858151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.857953    6777 generic.go:356] "Generic (PLEG): container finished" podID="43f115ae-b735-46cd-a150-2c525e2f325d" containerID="18080aa1aebcbd462bd5a3d7fc1997223b328717f3c0072c884bf7a9dfd1a122" exitCode=0
Oct 30 19:47:57.858151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.858092    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk" event={"ID":"43f115ae-b735-46cd-a150-2c525e2f325d","Type":"ContainerDied","Data":"18080aa1aebcbd462bd5a3d7fc1997223b328717f3c0072c884bf7a9dfd1a122"}
Oct 30 19:47:57.861306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.860291    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"
Oct 30 19:47:57.861306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.861134    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5" event={"ID":"e8401cdc-e5c5-4c7a-8f24-078287a136a7","Type":"ContainerDied","Data":"e0833d27ca7c8f20dc936c15e283b89df738b3ef49e605609e9e6356b5efa8ad"}
Oct 30 19:47:57.861306 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.861199    6777 scope.go:122] "RemoveContainer" containerID="460a65234e747931ccc758234e67dfde453038fd2728104d097d8e5bea0ec2e9"
Oct 30 19:47:57.868811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.868783    6777 generic.go:356] "Generic (PLEG): container finished" podID="0b305198-770f-4d2b-8f37-edccc157cddf" containerID="3b24869fd31e9a0e2cbc8cd9ae6884b08497350f5fa6979b3175053b52f8ca92" exitCode=0
Oct 30 19:47:57.869073 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.869037    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2" event={"ID":"0b305198-770f-4d2b-8f37-edccc157cddf","Type":"ContainerDied","Data":"3b24869fd31e9a0e2cbc8cd9ae6884b08497350f5fa6979b3175053b52f8ca92"}
Oct 30 19:47:57.875408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.875380    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:57.896262 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.896234    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:57.944838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.944800    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"]
Oct 30 19:47:57.948546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.948510    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b305198-770f-4d2b-8f37-edccc157cddf-kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/0b305198-770f-4d2b-8f37-edccc157cddf-kube-api-access-zn7mf\") pod \"0b305198-770f-4d2b-8f37-edccc157cddf\" (UID: \"0b305198-770f-4d2b-8f37-edccc157cddf\") "
Oct 30 19:47:57.948785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.948686    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f115ae-b735-46cd-a150-2c525e2f325d-kube-api-access-g9rnr\" (UniqueName: \"kubernetes.io/projected/43f115ae-b735-46cd-a150-2c525e2f325d-kube-api-access-g9rnr\") pod \"43f115ae-b735-46cd-a150-2c525e2f325d\" (UID: \"43f115ae-b735-46cd-a150-2c525e2f325d\") "
Oct 30 19:47:57.948785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.948740    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f115ae-b735-46cd-a150-2c525e2f325d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f115ae-b735-46cd-a150-2c525e2f325d-plugins-config-volume\") pod \"43f115ae-b735-46cd-a150-2c525e2f325d\" (UID: \"43f115ae-b735-46cd-a150-2c525e2f325d\") "
Oct 30 19:47:57.948785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.948768    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b305198-770f-4d2b-8f37-edccc157cddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b305198-770f-4d2b-8f37-edccc157cddf-plugins-config-volume\") pod \"0b305198-770f-4d2b-8f37-edccc157cddf\" (UID: \"0b305198-770f-4d2b-8f37-edccc157cddf\") "
Oct 30 19:47:57.949646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.949619    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b305198-770f-4d2b-8f37-edccc157cddf-plugins-config-volume" pod "0b305198-770f-4d2b-8f37-edccc157cddf" (UID: "0b305198-770f-4d2b-8f37-edccc157cddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:57.950347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.950310    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f115ae-b735-46cd-a150-2c525e2f325d-plugins-config-volume" pod "43f115ae-b735-46cd-a150-2c525e2f325d" (UID: "43f115ae-b735-46cd-a150-2c525e2f325d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:57.951955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.951934    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f115ae-b735-46cd-a150-2c525e2f325d-kube-api-access-g9rnr" pod "43f115ae-b735-46cd-a150-2c525e2f325d" (UID: "43f115ae-b735-46cd-a150-2c525e2f325d"). InnerVolumeSpecName "kube-api-access-g9rnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:57.952250 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.952231    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b305198-770f-4d2b-8f37-edccc157cddf-kube-api-access-zn7mf" pod "0b305198-770f-4d2b-8f37-edccc157cddf" (UID: "0b305198-770f-4d2b-8f37-edccc157cddf"). InnerVolumeSpecName "kube-api-access-zn7mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:57.955317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:57.955285    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1b1531-596d8b977f-z9mj5"]
Oct 30 19:47:58.051240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.051197    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9rnr\" (UniqueName: \"kubernetes.io/projected/43f115ae-b735-46cd-a150-2c525e2f325d-kube-api-access-g9rnr\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:58.051240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.051234    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f115ae-b735-46cd-a150-2c525e2f325d-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:58.051240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.051243    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b305198-770f-4d2b-8f37-edccc157cddf-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:58.051240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.051252    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/0b305198-770f-4d2b-8f37-edccc157cddf-kube-api-access-zn7mf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:58.134768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.134680    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8401cdc-e5c5-4c7a-8f24-078287a136a7" path="/var/lib/kubelet/pods/e8401cdc-e5c5-4c7a-8f24-078287a136a7/volumes"
Oct 30 19:47:58.613835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.613804    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:47:58.874373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.874278    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"]
Oct 30 19:47:58.874848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.874536    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf" podUID="f51d41eb-ec1d-4a77-9b41-a4bd92c26a38" containerName="payload-pre-processing" containerID="cri-o://f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138" gracePeriod=30
Oct 30 19:47:58.887558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.887207    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"
Oct 30 19:47:58.887558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.887380    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk" event={"ID":"43f115ae-b735-46cd-a150-2c525e2f325d","Type":"ContainerDied","Data":"eaa88a8d584053a66b108a7e926497af6afbe2f8fe463139bdd89d8de52028a5"}
Oct 30 19:47:58.887558 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.887462    6777 scope.go:122] "RemoveContainer" containerID="18080aa1aebcbd462bd5a3d7fc1997223b328717f3c0072c884bf7a9dfd1a122"
Oct 30 19:47:58.892686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.892643    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2" event={"ID":"0b305198-770f-4d2b-8f37-edccc157cddf","Type":"ContainerDied","Data":"8275976852d1d16c6669063c2d8007984602d527ef2d0d8b6f976c504a44399b"}
Oct 30 19:47:58.892829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.892774    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"
Oct 30 19:47:58.909045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.905905    6777 scope.go:122] "RemoveContainer" containerID="3b24869fd31e9a0e2cbc8cd9ae6884b08497350f5fa6979b3175053b52f8ca92"
Oct 30 19:47:58.944178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.944091    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"]
Oct 30 19:47:58.948783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.948744    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-74bbd9584d-nxjsk"]
Oct 30 19:47:58.963900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.963812    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"]
Oct 30 19:47:58.967698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:58.967662    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-6d7954cf9-tflx2"]
Oct 30 19:47:59.215294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.215194    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"]
Oct 30 19:47:59.218483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.218433    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-826518-nwzcz"]
Oct 30 19:47:59.359813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.359779    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:47:59.505456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.505360    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-kube-api-access-6z8vq\" (UniqueName: \"kubernetes.io/projected/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-kube-api-access-6z8vq\") pod \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\" (UID: \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\") "
Oct 30 19:47:59.505633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.505468    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-plugins-config-volume\") pod \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\" (UID: \"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38\") "
Oct 30 19:47:59.505850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.505822    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-plugins-config-volume" pod "f51d41eb-ec1d-4a77-9b41-a4bd92c26a38" (UID: "f51d41eb-ec1d-4a77-9b41-a4bd92c26a38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:59.505961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.505940    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:59.507659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.507635    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-kube-api-access-6z8vq" pod "f51d41eb-ec1d-4a77-9b41-a4bd92c26a38" (UID: "f51d41eb-ec1d-4a77-9b41-a4bd92c26a38"). InnerVolumeSpecName "kube-api-access-6z8vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:59.609562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.609517    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8vq\" (UniqueName: \"kubernetes.io/projected/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38-kube-api-access-6z8vq\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:59.629240 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.629192    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89" podUID="63b33f32-ffbe-4bbc-b34e-a439325206ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:59.915999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.915964    6777 generic.go:356] "Generic (PLEG): container finished" podID="f51d41eb-ec1d-4a77-9b41-a4bd92c26a38" containerID="f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138" exitCode=0
Oct 30 19:47:59.916527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.916177    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"
Oct 30 19:47:59.916527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.916201    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf" event={"ID":"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38","Type":"ContainerDied","Data":"f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138"}
Oct 30 19:47:59.916527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.916262    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf" event={"ID":"f51d41eb-ec1d-4a77-9b41-a4bd92c26a38","Type":"ContainerDied","Data":"524faa13c63703f7df4ae1c977c324b92f023255bce091526b64601ac513a7bb"}
Oct 30 19:47:59.916527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.916292    6777 scope.go:122] "RemoveContainer" containerID="f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138"
Oct 30 19:47:59.931792 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.931768    6777 scope.go:122] "RemoveContainer" containerID="f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138"
Oct 30 19:47:59.932222 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:47:59.932196    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138\": container with ID starting with f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138 not found: ID does not exist" containerID="f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138"
Oct 30 19:47:59.932351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.932332    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138"} err="failed to get container status \"f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138\": rpc error: code = NotFound desc = could not find container \"f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138\": container with ID starting with f675552fb3b9025200c5939765ec2e41ff914d9e18401aef8b34295e0c924138 not found: ID does not exist"
Oct 30 19:47:59.949143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.949108    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"]
Oct 30 19:47:59.957721 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:47:59.957624    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-826518-5879559bc5-tf7wf"]
Oct 30 19:48:00.114114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:00.114082    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b305198-770f-4d2b-8f37-edccc157cddf" path="/var/lib/kubelet/pods/0b305198-770f-4d2b-8f37-edccc157cddf/volumes"
Oct 30 19:48:00.114566 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:00.114552    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f115ae-b735-46cd-a150-2c525e2f325d" path="/var/lib/kubelet/pods/43f115ae-b735-46cd-a150-2c525e2f325d/volumes"
Oct 30 19:48:00.114974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:00.114963    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91275d29-3636-4502-9bff-c624dceebc77" path="/var/lib/kubelet/pods/91275d29-3636-4502-9bff-c624dceebc77/volumes"
Oct 30 19:48:00.115702 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:00.115686    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51d41eb-ec1d-4a77-9b41-a4bd92c26a38" path="/var/lib/kubelet/pods/f51d41eb-ec1d-4a77-9b41-a4bd92c26a38/volumes"
Oct 30 19:48:05.881952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:05.881917    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"]
Oct 30 19:48:05.882796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:05.882764    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy" containerID="cri-o://6adfd2d06f7a918124211ca69f6d52079608a98488f4d15f0a944956f54a0b0d" gracePeriod=30
Oct 30 19:48:05.886259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:05.886227    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"]
Oct 30 19:48:05.886508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:05.886473    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy" containerID="cri-o://16a2eaeaf9f8b4f67c93b39e5c09039e0ce85be048033c3a550ef923962b2a8c" gracePeriod=30
Oct 30 19:48:06.632132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:06.632098    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:06.632311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:06.632173    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:07.883080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:07.883032    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"]
Oct 30 19:48:07.883510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:07.883284    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy" containerID="cri-o://118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c" gracePeriod=30
Oct 30 19:48:08.404628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.404594    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"]
Oct 30 19:48:08.405221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405206    6777 cpu_manager.go:414] "containerMap: removing container" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy"
Oct 30 19:48:08.405221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405221    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405240    6777 cpu_manager.go:414] "containerMap: removing container" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405246    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405254    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405261    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405273    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e8401cdc-e5c5-4c7a-8f24-078287a136a7" containerName="payload-pre-processing"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405278    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8401cdc-e5c5-4c7a-8f24-078287a136a7" containerName="payload-pre-processing"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405285    6777 cpu_manager.go:414] "containerMap: removing container" podUID="43f115ae-b735-46cd-a150-2c525e2f325d" containerName="payload-processing"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405290    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f115ae-b735-46cd-a150-2c525e2f325d" containerName="payload-processing"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405299    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f51d41eb-ec1d-4a77-9b41-a4bd92c26a38" containerName="payload-pre-processing"
Oct 30 19:48:08.405330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405304    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51d41eb-ec1d-4a77-9b41-a4bd92c26a38" containerName="payload-pre-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405337    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cdffbd8e-5209-4b25-b923-92e782647e4b" containerName="payload-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405345    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdffbd8e-5209-4b25-b923-92e782647e4b" containerName="payload-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405353    6777 cpu_manager.go:414] "containerMap: removing container" podUID="91275d29-3636-4502-9bff-c624dceebc77" containerName="revoke-keys"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405358    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91275d29-3636-4502-9bff-c624dceebc77" containerName="revoke-keys"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405377    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0b305198-770f-4d2b-8f37-edccc157cddf" containerName="payload-pre-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405382    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b305198-770f-4d2b-8f37-edccc157cddf" containerName="payload-pre-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405389    6777 cpu_manager.go:414] "containerMap: removing container" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405393    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405399    6777 cpu_manager.go:414] "containerMap: removing container" podUID="4a4dd36a-3268-45c8-a86e-a37c70e77081" containerName="payload-pre-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405403    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4dd36a-3268-45c8-a86e-a37c70e77081" containerName="payload-pre-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405485    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdffbd8e-5209-4b25-b923-92e782647e4b" containerName="payload-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405493    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405500    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="91275d29-3636-4502-9bff-c624dceebc77" containerName="revoke-keys"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405505    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405514    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b305198-770f-4d2b-8f37-edccc157cddf" containerName="payload-pre-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405521    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="86401767-2d4b-430d-bfd1-11208842daca" containerName="istio-proxy"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405527    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4dd36a-3268-45c8-a86e-a37c70e77081" containerName="payload-pre-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405531    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a53f7aa-0c6e-4f0c-8e86-8d20cf6a9ea4" containerName="istio-proxy"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405539    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405546    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8401cdc-e5c5-4c7a-8f24-078287a136a7" containerName="payload-pre-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405552    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f115ae-b735-46cd-a150-2c525e2f325d" containerName="payload-processing"
Oct 30 19:48:08.405650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.405558    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51d41eb-ec1d-4a77-9b41-a4bd92c26a38" containerName="payload-pre-processing"
Oct 30 19:48:08.410794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.410775    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.415629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.415601    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:08.416867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.416846    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:08.423812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.423782    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"]
Oct 30 19:48:08.514992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.514949    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.515271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.515038    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.515361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.515332    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fczn\" (UniqueName: \"kubernetes.io/projected/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-kube-api-access-6fczn\") pod \"maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.616964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.616920    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fczn\" (UniqueName: \"kubernetes.io/projected/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-kube-api-access-6fczn\") pod \"maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.617169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.617017    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.617169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.617104    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.617698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.617674    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.619441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.619420    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.630036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.630006    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fczn\" (UniqueName: \"kubernetes.io/projected/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-kube-api-access-6fczn\") pod \"maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:08.726925 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:08.726826    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:09.011201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:09.010698    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"]
Oct 30 19:48:09.037074 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:48:09.037024    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b181cf_d297_4bcf_bf4f_2e0a7fd958cd.slice/crio-e5a473312cb9f6462d76451f3bef87c64d03ccefe8357fc40378b87cbaf52b03 WatchSource:0}: Error finding container e5a473312cb9f6462d76451f3bef87c64d03ccefe8357fc40378b87cbaf52b03: Status 404 returned error can't find the container with id e5a473312cb9f6462d76451f3bef87c64d03ccefe8357fc40378b87cbaf52b03
Oct 30 19:48:09.074307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:09.074273    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx" event={"ID":"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd","Type":"ContainerStarted","Data":"e5a473312cb9f6462d76451f3bef87c64d03ccefe8357fc40378b87cbaf52b03"}
Oct 30 19:48:09.602889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:09.602848    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89" podUID="63b33f32-ffbe-4bbc-b34e-a439325206ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:10.082807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:10.082770    6777 generic.go:356] "Generic (PLEG): container finished" podID="14b181cf-d297-4bcf-bf4f-2e0a7fd958cd" containerID="f2aace53d95d8511ea3b3363f8460f04c8334684e0e7a88e4646b58b45c5d55a" exitCode=0
Oct 30 19:48:10.083260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:10.082901    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx" event={"ID":"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd","Type":"ContainerDied","Data":"f2aace53d95d8511ea3b3363f8460f04c8334684e0e7a88e4646b58b45c5d55a"}
Oct 30 19:48:11.164154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.143343    6777 generic.go:356] "Generic (PLEG): container finished" podID="b9da534d-2d99-4e05-8866-6915937d2c14" containerID="16a2eaeaf9f8b4f67c93b39e5c09039e0ce85be048033c3a550ef923962b2a8c" exitCode=0
Oct 30 19:48:11.164154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.143635    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" event={"ID":"b9da534d-2d99-4e05-8866-6915937d2c14","Type":"ContainerDied","Data":"16a2eaeaf9f8b4f67c93b39e5c09039e0ce85be048033c3a550ef923962b2a8c"}
Oct 30 19:48:11.164154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.146254    6777 generic.go:356] "Generic (PLEG): container finished" podID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerID="6adfd2d06f7a918124211ca69f6d52079608a98488f4d15f0a944956f54a0b0d" exitCode=0
Oct 30 19:48:11.164154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.146525    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" event={"ID":"c53b1878-fc90-406e-bfcc-ac85764c7cec","Type":"ContainerDied","Data":"6adfd2d06f7a918124211ca69f6d52079608a98488f4d15f0a944956f54a0b0d"}
Oct 30 19:48:11.296732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.296680    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:48:11.328369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.328031    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:48:11.359369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.359333    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-istio-token\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-istio-token\") pod \"b9da534d-2d99-4e05-8866-6915937d2c14\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") "
Oct 30 19:48:11.359571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.359408    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53b1878-fc90-406e-bfcc-ac85764c7cec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53b1878-fc90-406e-bfcc-ac85764c7cec-istiod-ca-cert\") pod \"c53b1878-fc90-406e-bfcc-ac85764c7cec\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") "
Oct 30 19:48:11.359571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.359558    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9da534d-2d99-4e05-8866-6915937d2c14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da534d-2d99-4e05-8866-6915937d2c14-istio-podinfo\") pod \"b9da534d-2d99-4e05-8866-6915937d2c14\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") "
Oct 30 19:48:11.359708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.359634    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-certs\") pod \"b9da534d-2d99-4e05-8866-6915937d2c14\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") "
Oct 30 19:48:11.359768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.359742    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53b1878-fc90-406e-bfcc-ac85764c7cec-istiod-ca-cert" pod "c53b1878-fc90-406e-bfcc-ac85764c7cec" (UID: "c53b1878-fc90-406e-bfcc-ac85764c7cec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:11.360127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.360040    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-envoy\") pod \"c53b1878-fc90-406e-bfcc-ac85764c7cec\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") "
Oct 30 19:48:11.360293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.360129    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9da534d-2d99-4e05-8866-6915937d2c14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da534d-2d99-4e05-8866-6915937d2c14-istiod-ca-cert\") pod \"b9da534d-2d99-4e05-8866-6915937d2c14\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") "
Oct 30 19:48:11.360868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.360129    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-certs" pod "b9da534d-2d99-4e05-8866-6915937d2c14" (UID: "b9da534d-2d99-4e05-8866-6915937d2c14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.361012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.360656    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9da534d-2d99-4e05-8866-6915937d2c14-istiod-ca-cert" pod "b9da534d-2d99-4e05-8866-6915937d2c14" (UID: "b9da534d-2d99-4e05-8866-6915937d2c14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:11.361354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.361272    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53b1878-fc90-406e-bfcc-ac85764c7cec-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.361354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.361298    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.361354 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.361313    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9da534d-2d99-4e05-8866-6915937d2c14-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.362688 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.362651    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9da534d-2d99-4e05-8866-6915937d2c14-istio-podinfo" pod "b9da534d-2d99-4e05-8866-6915937d2c14" (UID: "b9da534d-2d99-4e05-8866-6915937d2c14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:11.364857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.364304    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-envoy" pod "c53b1878-fc90-406e-bfcc-ac85764c7cec" (UID: "c53b1878-fc90-406e-bfcc-ac85764c7cec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.364857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.364311    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-istio-token" pod "b9da534d-2d99-4e05-8866-6915937d2c14" (UID: "b9da534d-2d99-4e05-8866-6915937d2c14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.369352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.369327    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:11.377025 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.376994    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"]
Oct 30 19:48:11.462245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462155    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-credential-socket\") pod \"c53b1878-fc90-406e-bfcc-ac85764c7cec\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") "
Oct 30 19:48:11.462245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462237    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-kube-api-access-6fczn\" (UniqueName: \"kubernetes.io/projected/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-kube-api-access-6fczn\") pod \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") "
Oct 30 19:48:11.462495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462377    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-xks-ca\") pod \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") "
Oct 30 19:48:11.462495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462418    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-socket\") pod \"c53b1878-fc90-406e-bfcc-ac85764c7cec\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") "
Oct 30 19:48:11.462495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462454    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-credential-socket" pod "c53b1878-fc90-406e-bfcc-ac85764c7cec" (UID: "c53b1878-fc90-406e-bfcc-ac85764c7cec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.462495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462460    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-credential-socket\") pod \"b9da534d-2d99-4e05-8866-6915937d2c14\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") "
Oct 30 19:48:11.462729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462522    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-data\") pod \"c53b1878-fc90-406e-bfcc-ac85764c7cec\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") "
Oct 30 19:48:11.462729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462573    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-kube-api-access-6b8qm\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-kube-api-access-6b8qm\") pod \"c53b1878-fc90-406e-bfcc-ac85764c7cec\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") "
Oct 30 19:48:11.462729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462601    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-data\") pod \"b9da534d-2d99-4e05-8866-6915937d2c14\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") "
Oct 30 19:48:11.462729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462673    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-credential-socket" pod "b9da534d-2d99-4e05-8866-6915937d2c14" (UID: "b9da534d-2d99-4e05-8866-6915937d2c14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.462729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462677    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-socket" pod "c53b1878-fc90-406e-bfcc-ac85764c7cec" (UID: "c53b1878-fc90-406e-bfcc-ac85764c7cec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.462729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462706    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-kube-api-access-xlpmb\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-kube-api-access-xlpmb\") pod \"b9da534d-2d99-4e05-8866-6915937d2c14\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") "
Oct 30 19:48:11.463037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462737    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-podinfo\") pod \"c53b1878-fc90-406e-bfcc-ac85764c7cec\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") "
Oct 30 19:48:11.463037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462778    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-envoy\") pod \"b9da534d-2d99-4e05-8866-6915937d2c14\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") "
Oct 30 19:48:11.463037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462815    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-socket\") pod \"b9da534d-2d99-4e05-8866-6915937d2c14\" (UID: \"b9da534d-2d99-4e05-8866-6915937d2c14\") "
Oct 30 19:48:11.463037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462845    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-data" pod "c53b1878-fc90-406e-bfcc-ac85764c7cec" (UID: "c53b1878-fc90-406e-bfcc-ac85764c7cec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.463037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462852    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-openshift-ca\") pod \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\" (UID: \"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd\") "
Oct 30 19:48:11.463037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462899    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-certs\") pod \"c53b1878-fc90-406e-bfcc-ac85764c7cec\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") "
Oct 30 19:48:11.463037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.462931    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-token\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-token\") pod \"c53b1878-fc90-406e-bfcc-ac85764c7cec\" (UID: \"c53b1878-fc90-406e-bfcc-ac85764c7cec\") "
Oct 30 19:48:11.463553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.463211    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-data" pod "b9da534d-2d99-4e05-8866-6915937d2c14" (UID: "b9da534d-2d99-4e05-8866-6915937d2c14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.463553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.463324    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-certs" pod "c53b1878-fc90-406e-bfcc-ac85764c7cec" (UID: "c53b1878-fc90-406e-bfcc-ac85764c7cec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.463742 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.463720    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-socket" pod "b9da534d-2d99-4e05-8866-6915937d2c14" (UID: "b9da534d-2d99-4e05-8866-6915937d2c14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464614    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464643    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464659    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464673    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464687    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9da534d-2d99-4e05-8866-6915937d2c14-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464700    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464714    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464728    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464741    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.464754    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.465779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.465757    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-envoy" pod "b9da534d-2d99-4e05-8866-6915937d2c14" (UID: "b9da534d-2d99-4e05-8866-6915937d2c14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.465779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.465771    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-kube-api-access-xlpmb" pod "b9da534d-2d99-4e05-8866-6915937d2c14" (UID: "b9da534d-2d99-4e05-8866-6915937d2c14"). InnerVolumeSpecName "kube-api-access-xlpmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.466011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.465798    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-kube-api-access-6b8qm" pod "c53b1878-fc90-406e-bfcc-ac85764c7cec" (UID: "c53b1878-fc90-406e-bfcc-ac85764c7cec"). InnerVolumeSpecName "kube-api-access-6b8qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.466011 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.465813    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-kube-api-access-6fczn" pod "14b181cf-d297-4bcf-bf4f-2e0a7fd958cd" (UID: "14b181cf-d297-4bcf-bf4f-2e0a7fd958cd"). InnerVolumeSpecName "kube-api-access-6fczn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.466181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.466050    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-openshift-ca" pod "14b181cf-d297-4bcf-bf4f-2e0a7fd958cd" (UID: "14b181cf-d297-4bcf-bf4f-2e0a7fd958cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:11.466235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.466185    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-podinfo" pod "c53b1878-fc90-406e-bfcc-ac85764c7cec" (UID: "c53b1878-fc90-406e-bfcc-ac85764c7cec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:11.466763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.466737    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-token" pod "c53b1878-fc90-406e-bfcc-ac85764c7cec" (UID: "c53b1878-fc90-406e-bfcc-ac85764c7cec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.467850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.467817    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-xks-ca" pod "14b181cf-d297-4bcf-bf4f-2e0a7fd958cd" (UID: "14b181cf-d297-4bcf-bf4f-2e0a7fd958cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:11.566675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.566638    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.566675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.566673    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b8qm\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-kube-api-access-6b8qm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.566675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.566687    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlpmb\" (UniqueName: \"kubernetes.io/projected/b9da534d-2d99-4e05-8866-6915937d2c14-kube-api-access-xlpmb\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.566955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.566699    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.566955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.566712    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9da534d-2d99-4e05-8866-6915937d2c14-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.566955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.566724    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.566955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.566735    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53b1878-fc90-406e-bfcc-ac85764c7cec-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.566955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:11.566748    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fczn\" (UniqueName: \"kubernetes.io/projected/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd-kube-api-access-6fczn\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:12.153992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.153960    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"
Oct 30 19:48:12.154235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.153992    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx" event={"ID":"14b181cf-d297-4bcf-bf4f-2e0a7fd958cd","Type":"ContainerDied","Data":"e5a473312cb9f6462d76451f3bef87c64d03ccefe8357fc40378b87cbaf52b03"}
Oct 30 19:48:12.154235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.154052    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a473312cb9f6462d76451f3bef87c64d03ccefe8357fc40378b87cbaf52b03"
Oct 30 19:48:12.155658 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.155627    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"
Oct 30 19:48:12.155820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.155657    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd" event={"ID":"c53b1878-fc90-406e-bfcc-ac85764c7cec","Type":"ContainerDied","Data":"37acdab2f828a9d4cd134598d614f3381d724e42ceb273be7d074f538470b62b"}
Oct 30 19:48:12.155820 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.155710    6777 scope.go:122] "RemoveContainer" containerID="6adfd2d06f7a918124211ca69f6d52079608a98488f4d15f0a944956f54a0b0d"
Oct 30 19:48:12.157552 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.157537    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"
Oct 30 19:48:12.157654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.157572    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958" event={"ID":"b9da534d-2d99-4e05-8866-6915937d2c14","Type":"ContainerDied","Data":"40fe429c0f159e53056bdd38df5a478d590f97b4d152d7b90a48d9c1fb473b4f"}
Oct 30 19:48:12.169758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.169735    6777 scope.go:122] "RemoveContainer" containerID="16a2eaeaf9f8b4f67c93b39e5c09039e0ce85be048033c3a550ef923962b2a8c"
Oct 30 19:48:12.187970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.187940    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"]
Oct 30 19:48:12.194675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.194648    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-qz958"]
Oct 30 19:48:12.208711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.208680    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"]
Oct 30 19:48:12.214665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.214635    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-frcfd"]
Oct 30 19:48:12.633272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:12.633241    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"]
Oct 30 19:48:13.148021 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.147995    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:48:13.167348 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.167249    6777 generic.go:356] "Generic (PLEG): container finished" podID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerID="118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c" exitCode=0
Oct 30 19:48:13.167348 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.167309    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"
Oct 30 19:48:13.167591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.167337    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" event={"ID":"58823786-a7bd-4cc8-9a20-da28e6921be2","Type":"ContainerDied","Data":"118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c"}
Oct 30 19:48:13.167591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.167406    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr" event={"ID":"58823786-a7bd-4cc8-9a20-da28e6921be2","Type":"ContainerDied","Data":"856df76a20383de9072649b091932177053822b66ca02ab568c0f2b9aca49ad2"}
Oct 30 19:48:13.167591 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.167437    6777 scope.go:122] "RemoveContainer" containerID="118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c"
Oct 30 19:48:13.179108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.179033    6777 scope.go:122] "RemoveContainer" containerID="87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487"
Oct 30 19:48:13.189904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.189878    6777 scope.go:122] "RemoveContainer" containerID="118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c"
Oct 30 19:48:13.190416 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:13.190392    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c\": container with ID starting with 118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c not found: ID does not exist" containerID="118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c"
Oct 30 19:48:13.190548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.190427    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c"} err="failed to get container status \"118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c\": rpc error: code = NotFound desc = could not find container \"118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c\": container with ID starting with 118a73c33755533b7f526c19ad182aebb252e46f6ba5c14e9e33017fae4eb79c not found: ID does not exist"
Oct 30 19:48:13.190548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.190452    6777 scope.go:122] "RemoveContainer" containerID="87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487"
Oct 30 19:48:13.190548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.190499    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-socket\") pod \"58823786-a7bd-4cc8-9a20-da28e6921be2\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") "
Oct 30 19:48:13.190709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.190581    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-credential-socket\") pod \"58823786-a7bd-4cc8-9a20-da28e6921be2\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") "
Oct 30 19:48:13.190763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.190748    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-socket" pod "58823786-a7bd-4cc8-9a20-da28e6921be2" (UID: "58823786-a7bd-4cc8-9a20-da28e6921be2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:13.190960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.190942    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-credential-socket" pod "58823786-a7bd-4cc8-9a20-da28e6921be2" (UID: "58823786-a7bd-4cc8-9a20-da28e6921be2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:13.191091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.190686    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-podinfo\") pod \"58823786-a7bd-4cc8-9a20-da28e6921be2\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") "
Oct 30 19:48:13.191248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.191206    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-token\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-token\") pod \"58823786-a7bd-4cc8-9a20-da28e6921be2\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") "
Oct 30 19:48:13.191351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.191270    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58823786-a7bd-4cc8-9a20-da28e6921be2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58823786-a7bd-4cc8-9a20-da28e6921be2-istiod-ca-cert\") pod \"58823786-a7bd-4cc8-9a20-da28e6921be2\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") "
Oct 30 19:48:13.191351 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.191335    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-envoy\") pod \"58823786-a7bd-4cc8-9a20-da28e6921be2\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") "
Oct 30 19:48:13.191438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.191361    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-certs\") pod \"58823786-a7bd-4cc8-9a20-da28e6921be2\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") "
Oct 30 19:48:13.191438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.191385    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-data\") pod \"58823786-a7bd-4cc8-9a20-da28e6921be2\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") "
Oct 30 19:48:13.191751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.191534    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-kube-api-access-kn2mq\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-kube-api-access-kn2mq\") pod \"58823786-a7bd-4cc8-9a20-da28e6921be2\" (UID: \"58823786-a7bd-4cc8-9a20-da28e6921be2\") "
Oct 30 19:48:13.191751 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:13.191591    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487\": container with ID starting with 87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487 not found: ID does not exist" containerID="87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487"
Oct 30 19:48:13.191751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.191621    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487"} err="failed to get container status \"87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487\": rpc error: code = NotFound desc = could not find container \"87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487\": container with ID starting with 87b8cc30daf3a295d5a119731b202f2f1d48e403a01fd8a186bc38f6fd7ce487 not found: ID does not exist"
Oct 30 19:48:13.192198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.192173    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:13.192295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.192222    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:13.192521 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.192495    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-data" pod "58823786-a7bd-4cc8-9a20-da28e6921be2" (UID: "58823786-a7bd-4cc8-9a20-da28e6921be2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:13.192683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.192662    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58823786-a7bd-4cc8-9a20-da28e6921be2-istiod-ca-cert" pod "58823786-a7bd-4cc8-9a20-da28e6921be2" (UID: "58823786-a7bd-4cc8-9a20-da28e6921be2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:13.192830 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.192776    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-certs" pod "58823786-a7bd-4cc8-9a20-da28e6921be2" (UID: "58823786-a7bd-4cc8-9a20-da28e6921be2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:13.194013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.193979    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-token" pod "58823786-a7bd-4cc8-9a20-da28e6921be2" (UID: "58823786-a7bd-4cc8-9a20-da28e6921be2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:13.194128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.194096    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-podinfo" pod "58823786-a7bd-4cc8-9a20-da28e6921be2" (UID: "58823786-a7bd-4cc8-9a20-da28e6921be2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:13.194493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.194466    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-kube-api-access-kn2mq" pod "58823786-a7bd-4cc8-9a20-da28e6921be2" (UID: "58823786-a7bd-4cc8-9a20-da28e6921be2"). InnerVolumeSpecName "kube-api-access-kn2mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:13.194741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.194716    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-envoy" pod "58823786-a7bd-4cc8-9a20-da28e6921be2" (UID: "58823786-a7bd-4cc8-9a20-da28e6921be2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:13.293402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.293362    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:13.293402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.293394    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:13.293638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.293424    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58823786-a7bd-4cc8-9a20-da28e6921be2-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:13.293638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.293436    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:13.293638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.293444    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:13.293638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.293453    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58823786-a7bd-4cc8-9a20-da28e6921be2-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:13.293638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.293462    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn2mq\" (UniqueName: \"kubernetes.io/projected/58823786-a7bd-4cc8-9a20-da28e6921be2-kube-api-access-kn2mq\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:13.495201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.495163    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"]
Oct 30 19:48:13.501175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.501141    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-826518-openshift-default-67bd854bff-5wppr"]
Oct 30 19:48:13.640504 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:13.640459    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:48:13.937019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.936982    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"]
Oct 30 19:48:13.937362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:13.937330    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x" podUID="a234178b-2e51-465e-9467-6f210c30799f" containerName="maas-api" containerID="cri-o://81af924074e7e171bde928f49afc7f1d6fec306305ed80df98772c9f0d794be8" gracePeriod=30
Oct 30 19:48:14.113988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.113949    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" path="/var/lib/kubelet/pods/58823786-a7bd-4cc8-9a20-da28e6921be2/volumes"
Oct 30 19:48:14.114678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.114649    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" path="/var/lib/kubelet/pods/b9da534d-2d99-4e05-8866-6915937d2c14/volumes"
Oct 30 19:48:14.115188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.115166    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" path="/var/lib/kubelet/pods/c53b1878-fc90-406e-bfcc-ac85764c7cec/volumes"
Oct 30 19:48:14.179794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.179741    6777 generic.go:356] "Generic (PLEG): container finished" podID="a234178b-2e51-465e-9467-6f210c30799f" containerID="81af924074e7e171bde928f49afc7f1d6fec306305ed80df98772c9f0d794be8" exitCode=0
Oct 30 19:48:14.180386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.179816    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x" event={"ID":"a234178b-2e51-465e-9467-6f210c30799f","Type":"ContainerDied","Data":"81af924074e7e171bde928f49afc7f1d6fec306305ed80df98772c9f0d794be8"}
Oct 30 19:48:14.237320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.237287    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:48:14.340638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.335978    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:48:14.439082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.439018    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-metrics-certs\") pod \"a234178b-2e51-465e-9467-6f210c30799f\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") "
Oct 30 19:48:14.439301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.439099    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-maas-api-tls\") pod \"a234178b-2e51-465e-9467-6f210c30799f\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") "
Oct 30 19:48:14.439301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.439253    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a234178b-2e51-465e-9467-6f210c30799f-kube-api-access-75mmt\" (UniqueName: \"kubernetes.io/projected/a234178b-2e51-465e-9467-6f210c30799f-kube-api-access-75mmt\") pod \"a234178b-2e51-465e-9467-6f210c30799f\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") "
Oct 30 19:48:14.439508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.439485    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a234178b-2e51-465e-9467-6f210c30799f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a234178b-2e51-465e-9467-6f210c30799f-openshift-service-ca\") pod \"a234178b-2e51-465e-9467-6f210c30799f\" (UID: \"a234178b-2e51-465e-9467-6f210c30799f\") "
Oct 30 19:48:14.439926 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.439889    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a234178b-2e51-465e-9467-6f210c30799f-openshift-service-ca" pod "a234178b-2e51-465e-9467-6f210c30799f" (UID: "a234178b-2e51-465e-9467-6f210c30799f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:14.440274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.440247    6777 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a234178b-2e51-465e-9467-6f210c30799f-openshift-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:14.441489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.441451    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-metrics-certs" pod "a234178b-2e51-465e-9467-6f210c30799f" (UID: "a234178b-2e51-465e-9467-6f210c30799f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:14.441785 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.441763    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a234178b-2e51-465e-9467-6f210c30799f-kube-api-access-75mmt" pod "a234178b-2e51-465e-9467-6f210c30799f" (UID: "a234178b-2e51-465e-9467-6f210c30799f"). InnerVolumeSpecName "kube-api-access-75mmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:14.441871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.441787    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-maas-api-tls" pod "a234178b-2e51-465e-9467-6f210c30799f" (UID: "a234178b-2e51-465e-9467-6f210c30799f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:14.541268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.541232    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75mmt\" (UniqueName: \"kubernetes.io/projected/a234178b-2e51-465e-9467-6f210c30799f-kube-api-access-75mmt\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:14.541268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.541265    6777 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-metrics-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:14.541268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:14.541275    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a234178b-2e51-465e-9467-6f210c30799f-maas-api-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.190245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:15.190205    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x" event={"ID":"a234178b-2e51-465e-9467-6f210c30799f","Type":"ContainerDied","Data":"55e5b9bb29c71ea3d7e1d06aded0e379be526f3eb50a14cd2e68046139797b32"}
Oct 30 19:48:15.190733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:15.190274    6777 scope.go:122] "RemoveContainer" containerID="81af924074e7e171bde928f49afc7f1d6fec306305ed80df98772c9f0d794be8"
Oct 30 19:48:15.190733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:15.190221    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"
Oct 30 19:48:15.214465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:15.214432    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"]
Oct 30 19:48:15.218724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:15.218693    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b281d5-577fddc6cf-rmn8x"]
Oct 30 19:48:16.113341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:16.113306    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a234178b-2e51-465e-9467-6f210c30799f" path="/var/lib/kubelet/pods/a234178b-2e51-465e-9467-6f210c30799f/volumes"
Oct 30 19:48:16.863991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:16.863964    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:48:16.874589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:16.874554    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b33f32-ffbe-4bbc-b34e-a439325206ab-kube-api-access-gsjsm\" (UniqueName: \"kubernetes.io/projected/63b33f32-ffbe-4bbc-b34e-a439325206ab-kube-api-access-gsjsm\") pod \"63b33f32-ffbe-4bbc-b34e-a439325206ab\" (UID: \"63b33f32-ffbe-4bbc-b34e-a439325206ab\") "
Oct 30 19:48:16.874802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:16.874695    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b33f32-ffbe-4bbc-b34e-a439325206ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b33f32-ffbe-4bbc-b34e-a439325206ab-plugins-config-volume\") pod \"63b33f32-ffbe-4bbc-b34e-a439325206ab\" (UID: \"63b33f32-ffbe-4bbc-b34e-a439325206ab\") "
Oct 30 19:48:16.875151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:16.875123    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b33f32-ffbe-4bbc-b34e-a439325206ab-plugins-config-volume" pod "63b33f32-ffbe-4bbc-b34e-a439325206ab" (UID: "63b33f32-ffbe-4bbc-b34e-a439325206ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:16.875297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:16.875282    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b33f32-ffbe-4bbc-b34e-a439325206ab-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:16.876711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:16.876683    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b33f32-ffbe-4bbc-b34e-a439325206ab-kube-api-access-gsjsm" pod "63b33f32-ffbe-4bbc-b34e-a439325206ab" (UID: "63b33f32-ffbe-4bbc-b34e-a439325206ab"). InnerVolumeSpecName "kube-api-access-gsjsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:17.020362 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:16.977285    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsjsm\" (UniqueName: \"kubernetes.io/projected/63b33f32-ffbe-4bbc-b34e-a439325206ab-kube-api-access-gsjsm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:17.206527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:17.206426    6777 generic.go:356] "Generic (PLEG): container finished" podID="63b33f32-ffbe-4bbc-b34e-a439325206ab" containerID="5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6" exitCode=0
Oct 30 19:48:17.206527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:17.206472    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"
Oct 30 19:48:17.206527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:17.206508    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89" event={"ID":"63b33f32-ffbe-4bbc-b34e-a439325206ab","Type":"ContainerDied","Data":"5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6"}
Oct 30 19:48:17.206856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:17.206569    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89" event={"ID":"63b33f32-ffbe-4bbc-b34e-a439325206ab","Type":"ContainerDied","Data":"e101cd10030b805c1ad0002c90447ff7005d5d131d9481d103b4faaf8e713cf2"}
Oct 30 19:48:17.206856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:17.206599    6777 scope.go:122] "RemoveContainer" containerID="5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6"
Oct 30 19:48:17.218866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:17.218844    6777 scope.go:122] "RemoveContainer" containerID="5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6"
Oct 30 19:48:17.219416 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:17.219379    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6\": container with ID starting with 5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6 not found: ID does not exist" containerID="5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6"
Oct 30 19:48:17.219546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:17.219428    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6"} err="failed to get container status \"5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6\": rpc error: code = NotFound desc = could not find container \"5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6\": container with ID starting with 5fea0e82eab103c312493cc459f413a05dcafc30930a15bd2aa14a13356cbae6 not found: ID does not exist"
Oct 30 19:48:17.236120 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:17.236083    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"]
Oct 30 19:48:17.240166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:17.240137    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1b1531-7d7cf7c59b-f7r89"]
Oct 30 19:48:18.113638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:18.113594    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b33f32-ffbe-4bbc-b34e-a439325206ab" path="/var/lib/kubelet/pods/63b33f32-ffbe-4bbc-b34e-a439325206ab/volumes"
Oct 30 19:48:19.203883 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.203844    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"]
Oct 30 19:48:19.204710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204686    6777 cpu_manager.go:414] "containerMap: removing container" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy"
Oct 30 19:48:19.204828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204711    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy"
Oct 30 19:48:19.204828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204725    6777 cpu_manager.go:414] "containerMap: removing container" podUID="a234178b-2e51-465e-9467-6f210c30799f" containerName="maas-api"
Oct 30 19:48:19.204828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204732    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a234178b-2e51-465e-9467-6f210c30799f" containerName="maas-api"
Oct 30 19:48:19.204828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204772    6777 cpu_manager.go:414] "containerMap: removing container" podUID="14b181cf-d297-4bcf-bf4f-2e0a7fd958cd" containerName="revoke-keys"
Oct 30 19:48:19.204828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204781    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b181cf-d297-4bcf-bf4f-2e0a7fd958cd" containerName="revoke-keys"
Oct 30 19:48:19.204828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204818    6777 cpu_manager.go:414] "containerMap: removing container" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy"
Oct 30 19:48:19.204828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204825    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85923a0f-3daa-4295-9a1d-3ca15689f158" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204838    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204844    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204856    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204864    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204883    6777 cpu_manager.go:414] "containerMap: removing container" podUID="63b33f32-ffbe-4bbc-b34e-a439325206ab" containerName="payload-processing"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204890    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b33f32-ffbe-4bbc-b34e-a439325206ab" containerName="payload-processing"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204904    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.204912    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.205086    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b181cf-d297-4bcf-bf4f-2e0a7fd958cd" containerName="revoke-keys"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.205109    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9da534d-2d99-4e05-8866-6915937d2c14" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.205122    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b33f32-ffbe-4bbc-b34e-a439325206ab" containerName="payload-processing"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.205133    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53b1878-fc90-406e-bfcc-ac85764c7cec" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.205145    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.205154    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.205163    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="a234178b-2e51-465e-9467-6f210c30799f" containerName="maas-api"
Oct 30 19:48:19.205234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.205173    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8d5446-5b96-41f1-815e-88c0d09b24a0" containerName="istio-proxy"
Oct 30 19:48:19.210623 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.210598    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.218904 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.218876    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"]
Oct 30 19:48:19.272701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.272658    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af8b9bed-wcrsd\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.272939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.272709    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af8b9bed-wcrsd\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.273018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.272987    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz62x\" (UniqueName: \"kubernetes.io/projected/df954591-84c1-4e39-89a9-2eb186ed69fa-kube-api-access-jz62x\") pod \"maas-api-revoke-sub-af8b9bed-wcrsd\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.315864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.315780    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"]
Oct 30 19:48:19.316255 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.316232    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:19.316365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.316292    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:19.316365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.316346    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:19.316465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.316387    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:19.316606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.316588    6777 cpu_manager.go:414] "containerMap: removing container" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy"
Oct 30 19:48:19.316652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.316607    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58823786-a7bd-4cc8-9a20-da28e6921be2" containerName="istio-proxy"
Oct 30 19:48:19.320349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.320327    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.341821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.341789    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"]
Oct 30 19:48:19.375040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.374999    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz62x\" (UniqueName: \"kubernetes.io/projected/df954591-84c1-4e39-89a9-2eb186ed69fa-kube-api-access-jz62x\") pod \"maas-api-revoke-sub-af8b9bed-wcrsd\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.375358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.375331    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f176a999-5q76s\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.375649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.375626    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af8b9bed-wcrsd\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.375775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.375759    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65vxg\" (UniqueName: \"kubernetes.io/projected/cbd9d42f-365e-451b-8d03-daca355eb025-kube-api-access-65vxg\") pod \"maas-api-revoke-sub-f176a999-5q76s\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.375917 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.375899    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af8b9bed-wcrsd\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.376136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.376115    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f176a999-5q76s\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.376424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.376400    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af8b9bed-wcrsd\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.379562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.379540    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af8b9bed-wcrsd\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.397375 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.397345    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz62x\" (UniqueName: \"kubernetes.io/projected/df954591-84c1-4e39-89a9-2eb186ed69fa-kube-api-access-jz62x\") pod \"maas-api-revoke-sub-af8b9bed-wcrsd\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.478044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.477920    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f176a999-5q76s\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.478253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.478123    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65vxg\" (UniqueName: \"kubernetes.io/projected/cbd9d42f-365e-451b-8d03-daca355eb025-kube-api-access-65vxg\") pod \"maas-api-revoke-sub-f176a999-5q76s\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.478253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.478223    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f176a999-5q76s\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.478674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.478654    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f176a999-5q76s\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.480690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.480663    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f176a999-5q76s\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.491740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.491709    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65vxg\" (UniqueName: \"kubernetes.io/projected/cbd9d42f-365e-451b-8d03-daca355eb025-kube-api-access-65vxg\") pod \"maas-api-revoke-sub-f176a999-5q76s\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.534452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.534407    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:19.635928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.634518    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:19.830141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.830108    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"]
Oct 30 19:48:19.831817 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:48:19.831787    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf954591_84c1_4e39_89a9_2eb186ed69fa.slice/crio-aa9665e72f06dad0c3037e2ca7624530e7911cf84c6f82022574138d0be691ff WatchSource:0}: Error finding container aa9665e72f06dad0c3037e2ca7624530e7911cf84c6f82022574138d0be691ff: Status 404 returned error can't find the container with id aa9665e72f06dad0c3037e2ca7624530e7911cf84c6f82022574138d0be691ff
Oct 30 19:48:19.976621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:19.976575    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"]
Oct 30 19:48:19.980767 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:48:19.980733    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd9d42f_365e_451b_8d03_daca355eb025.slice/crio-36c288549a5fb4e8186b3ad42ab4e0507d7d8034fd0c455e938611dc2578be87 WatchSource:0}: Error finding container 36c288549a5fb4e8186b3ad42ab4e0507d7d8034fd0c455e938611dc2578be87: Status 404 returned error can't find the container with id 36c288549a5fb4e8186b3ad42ab4e0507d7d8034fd0c455e938611dc2578be87
Oct 30 19:48:20.267793 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:20.267762    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s" event={"ID":"cbd9d42f-365e-451b-8d03-daca355eb025","Type":"ContainerStarted","Data":"36c288549a5fb4e8186b3ad42ab4e0507d7d8034fd0c455e938611dc2578be87"}
Oct 30 19:48:20.269538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:20.269511    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd" event={"ID":"df954591-84c1-4e39-89a9-2eb186ed69fa","Type":"ContainerStarted","Data":"86aa1ff670fe428b9bf2f5d97e362c11a2715bfacdf1f3c5ddb83d04ca1094f6"}
Oct 30 19:48:20.269667 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:20.269559    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd" event={"ID":"df954591-84c1-4e39-89a9-2eb186ed69fa","Type":"ContainerStarted","Data":"aa9665e72f06dad0c3037e2ca7624530e7911cf84c6f82022574138d0be691ff"}
Oct 30 19:48:20.290621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:20.290561    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd" podStartSLOduration=1.290544985 podStartE2EDuration="1.290544985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:20.28795886 +0000 UTC m=+2482.284874257" watchObservedRunningTime="2026-10-30 19:48:20.290544985 +0000 UTC m=+2482.287460383"
Oct 30 19:48:21.277218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:21.277179    6777 generic.go:356] "Generic (PLEG): container finished" podID="cbd9d42f-365e-451b-8d03-daca355eb025" containerID="8aefbd41eb22263aa92443582e9ca8a4e23babbef91f48845a78d934450369f9" exitCode=0
Oct 30 19:48:21.277694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:21.277264    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s" event={"ID":"cbd9d42f-365e-451b-8d03-daca355eb025","Type":"ContainerDied","Data":"8aefbd41eb22263aa92443582e9ca8a4e23babbef91f48845a78d934450369f9"}
Oct 30 19:48:21.279104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:21.279037    6777 generic.go:356] "Generic (PLEG): container finished" podID="df954591-84c1-4e39-89a9-2eb186ed69fa" containerID="86aa1ff670fe428b9bf2f5d97e362c11a2715bfacdf1f3c5ddb83d04ca1094f6" exitCode=0
Oct 30 19:48:21.279244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:21.279140    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd" event={"ID":"df954591-84c1-4e39-89a9-2eb186ed69fa","Type":"ContainerDied","Data":"86aa1ff670fe428b9bf2f5d97e362c11a2715bfacdf1f3c5ddb83d04ca1094f6"}
Oct 30 19:48:22.493107 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.493081    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:22.496472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.496453    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:22.636220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.636103    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df954591-84c1-4e39-89a9-2eb186ed69fa-kube-api-access-jz62x\" (UniqueName: \"kubernetes.io/projected/df954591-84c1-4e39-89a9-2eb186ed69fa-kube-api-access-jz62x\") pod \"df954591-84c1-4e39-89a9-2eb186ed69fa\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") "
Oct 30 19:48:22.636220 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.636214    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd9d42f-365e-451b-8d03-daca355eb025-kube-api-access-65vxg\" (UniqueName: \"kubernetes.io/projected/cbd9d42f-365e-451b-8d03-daca355eb025-kube-api-access-65vxg\") pod \"cbd9d42f-365e-451b-8d03-daca355eb025\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") "
Oct 30 19:48:22.638564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.636984    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-xks-ca\") pod \"cbd9d42f-365e-451b-8d03-daca355eb025\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") "
Oct 30 19:48:22.638564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.637867    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-xks-ca\") pod \"df954591-84c1-4e39-89a9-2eb186ed69fa\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") "
Oct 30 19:48:22.638564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.637913    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-openshift-ca\") pod \"df954591-84c1-4e39-89a9-2eb186ed69fa\" (UID: \"df954591-84c1-4e39-89a9-2eb186ed69fa\") "
Oct 30 19:48:22.638564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.637960    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-openshift-ca\") pod \"cbd9d42f-365e-451b-8d03-daca355eb025\" (UID: \"cbd9d42f-365e-451b-8d03-daca355eb025\") "
Oct 30 19:48:22.640791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.638951    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-openshift-ca" pod "cbd9d42f-365e-451b-8d03-daca355eb025" (UID: "cbd9d42f-365e-451b-8d03-daca355eb025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:22.640791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.639281    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-openshift-ca" pod "df954591-84c1-4e39-89a9-2eb186ed69fa" (UID: "df954591-84c1-4e39-89a9-2eb186ed69fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:22.640791 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.639708    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df954591-84c1-4e39-89a9-2eb186ed69fa-kube-api-access-jz62x" pod "df954591-84c1-4e39-89a9-2eb186ed69fa" (UID: "df954591-84c1-4e39-89a9-2eb186ed69fa"). InnerVolumeSpecName "kube-api-access-jz62x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:22.641461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.641433    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-xks-ca" pod "df954591-84c1-4e39-89a9-2eb186ed69fa" (UID: "df954591-84c1-4e39-89a9-2eb186ed69fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:22.641800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.641751    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd9d42f-365e-451b-8d03-daca355eb025-kube-api-access-65vxg" pod "cbd9d42f-365e-451b-8d03-daca355eb025" (UID: "cbd9d42f-365e-451b-8d03-daca355eb025"). InnerVolumeSpecName "kube-api-access-65vxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:22.642915 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.642893    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-xks-ca" pod "cbd9d42f-365e-451b-8d03-daca355eb025" (UID: "cbd9d42f-365e-451b-8d03-daca355eb025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:22.740664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.740611    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.740664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.740649    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.740664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.740664    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df954591-84c1-4e39-89a9-2eb186ed69fa-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.741199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.740680    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbd9d42f-365e-451b-8d03-daca355eb025-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.741199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.740693    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz62x\" (UniqueName: \"kubernetes.io/projected/df954591-84c1-4e39-89a9-2eb186ed69fa-kube-api-access-jz62x\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.741199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.740706    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65vxg\" (UniqueName: \"kubernetes.io/projected/cbd9d42f-365e-451b-8d03-daca355eb025-kube-api-access-65vxg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.752770 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.752735    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"]
Oct 30 19:48:22.753006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.752985    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj" podUID="0b018b2c-5358-4414-b836-baa46bc655e3" containerName="payload-pre-processing" containerID="cri-o://3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6" gracePeriod=30
Oct 30 19:48:22.975023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.974971    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"]
Oct 30 19:48:22.975248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:22.975152    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-be6b74-sjvqx"]
Oct 30 19:48:23.027991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.027959    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:48:23.145448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.145415    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b018b2c-5358-4414-b836-baa46bc655e3-kube-api-access-2lj95\" (UniqueName: \"kubernetes.io/projected/0b018b2c-5358-4414-b836-baa46bc655e3-kube-api-access-2lj95\") pod \"0b018b2c-5358-4414-b836-baa46bc655e3\" (UID: \"0b018b2c-5358-4414-b836-baa46bc655e3\") "
Oct 30 19:48:23.145642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.145572    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b018b2c-5358-4414-b836-baa46bc655e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b018b2c-5358-4414-b836-baa46bc655e3-plugins-config-volume\") pod \"0b018b2c-5358-4414-b836-baa46bc655e3\" (UID: \"0b018b2c-5358-4414-b836-baa46bc655e3\") "
Oct 30 19:48:23.145938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.145911    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b018b2c-5358-4414-b836-baa46bc655e3-plugins-config-volume" pod "0b018b2c-5358-4414-b836-baa46bc655e3" (UID: "0b018b2c-5358-4414-b836-baa46bc655e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:23.146159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.146134    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b018b2c-5358-4414-b836-baa46bc655e3-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:23.147512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.147489    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b018b2c-5358-4414-b836-baa46bc655e3-kube-api-access-2lj95" pod "0b018b2c-5358-4414-b836-baa46bc655e3" (UID: "0b018b2c-5358-4414-b836-baa46bc655e3"). InnerVolumeSpecName "kube-api-access-2lj95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:23.262304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.253334    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lj95\" (UniqueName: \"kubernetes.io/projected/0b018b2c-5358-4414-b836-baa46bc655e3-kube-api-access-2lj95\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:23.329890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.317834    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"]
Oct 30 19:48:23.329890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.318101    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy" containerID="cri-o://50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a" gracePeriod=30
Oct 30 19:48:23.361121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.361023    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"
Oct 30 19:48:23.361458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.361429    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s" event={"ID":"cbd9d42f-365e-451b-8d03-daca355eb025","Type":"ContainerDied","Data":"36c288549a5fb4e8186b3ad42ab4e0507d7d8034fd0c455e938611dc2578be87"}
Oct 30 19:48:23.361689 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.361673    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36c288549a5fb4e8186b3ad42ab4e0507d7d8034fd0c455e938611dc2578be87"
Oct 30 19:48:23.369032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.366815    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd" event={"ID":"df954591-84c1-4e39-89a9-2eb186ed69fa","Type":"ContainerDied","Data":"aa9665e72f06dad0c3037e2ca7624530e7911cf84c6f82022574138d0be691ff"}
Oct 30 19:48:23.369032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.366899    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa9665e72f06dad0c3037e2ca7624530e7911cf84c6f82022574138d0be691ff"
Oct 30 19:48:23.369032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.366967    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"
Oct 30 19:48:23.369032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.368883    6777 generic.go:356] "Generic (PLEG): container finished" podID="0b018b2c-5358-4414-b836-baa46bc655e3" containerID="3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6" exitCode=0
Oct 30 19:48:23.369032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.368963    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj" event={"ID":"0b018b2c-5358-4414-b836-baa46bc655e3","Type":"ContainerDied","Data":"3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6"}
Oct 30 19:48:23.369032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.369006    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj" event={"ID":"0b018b2c-5358-4414-b836-baa46bc655e3","Type":"ContainerDied","Data":"1dd50a3094cb0977ca21fbb5b7468e878d1d65401c44046e3dcc8a1efe34e4d0"}
Oct 30 19:48:23.369032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.369036    6777 scope.go:122] "RemoveContainer" containerID="3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6"
Oct 30 19:48:23.369824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.369207    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"
Oct 30 19:48:23.469525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.469015    6777 scope.go:122] "RemoveContainer" containerID="3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6"
Oct 30 19:48:23.482481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.482394    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"]
Oct 30 19:48:23.488912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.488842    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-5cdbb499ff-jmntj"]
Oct 30 19:48:23.489524 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:23.489372    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6\": container with ID starting with 3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6 not found: ID does not exist" containerID="3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6"
Oct 30 19:48:23.489729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.489701    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6"} err="failed to get container status \"3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6\": rpc error: code = NotFound desc = could not find container \"3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6\": container with ID starting with 3fc66a4921510aea22f3e0d63e0447d05b883fbb498e04e02a6b77681f767ea6 not found: ID does not exist"
Oct 30 19:48:23.786506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.786465    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"]
Oct 30 19:48:23.787307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.787284    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0b018b2c-5358-4414-b836-baa46bc655e3" containerName="payload-pre-processing"
Oct 30 19:48:23.787412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.787308    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b018b2c-5358-4414-b836-baa46bc655e3" containerName="payload-pre-processing"
Oct 30 19:48:23.787412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.787330    6777 cpu_manager.go:414] "containerMap: removing container" podUID="df954591-84c1-4e39-89a9-2eb186ed69fa" containerName="revoke-keys"
Oct 30 19:48:23.787412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.787338    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df954591-84c1-4e39-89a9-2eb186ed69fa" containerName="revoke-keys"
Oct 30 19:48:23.787412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.787357    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cbd9d42f-365e-451b-8d03-daca355eb025" containerName="revoke-keys"
Oct 30 19:48:23.787412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.787365    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd9d42f-365e-451b-8d03-daca355eb025" containerName="revoke-keys"
Oct 30 19:48:23.787678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.787547    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b018b2c-5358-4414-b836-baa46bc655e3" containerName="payload-pre-processing"
Oct 30 19:48:23.787678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.787567    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="df954591-84c1-4e39-89a9-2eb186ed69fa" containerName="revoke-keys"
Oct 30 19:48:23.787678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.787579    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd9d42f-365e-451b-8d03-daca355eb025" containerName="revoke-keys"
Oct 30 19:48:23.792268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.792235    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:23.798656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.798357    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:23.798656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.798483    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:23.799438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.799410    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"]
Oct 30 19:48:23.840092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.840042    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:23.840303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.840214    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:23.871173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.871138    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:23.871373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.871286    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:23.871463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.871446    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c59n\" (UniqueName: \"kubernetes.io/projected/ab296ad3-b845-4bd9-a48c-492cd1469ddd-kube-api-access-9c59n\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:23.972899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.972860    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c59n\" (UniqueName: \"kubernetes.io/projected/ab296ad3-b845-4bd9-a48c-492cd1469ddd-kube-api-access-9c59n\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:23.973132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.972952    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:23.973132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.973015    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:23.973690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.973666    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:23.975401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.975381    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:23.985617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:23.985580    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c59n\" (UniqueName: \"kubernetes.io/projected/ab296ad3-b845-4bd9-a48c-492cd1469ddd-kube-api-access-9c59n\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:24.113647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:24.113562    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b018b2c-5358-4414-b836-baa46bc655e3" path="/var/lib/kubelet/pods/0b018b2c-5358-4414-b836-baa46bc655e3/volumes"
Oct 30 19:48:24.114039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:24.114025    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b181cf-d297-4bcf-bf4f-2e0a7fd958cd" path="/var/lib/kubelet/pods/14b181cf-d297-4bcf-bf4f-2e0a7fd958cd/volumes"
Oct 30 19:48:24.136867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:24.136832    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:24.419033 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:24.418997    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"]
Oct 30 19:48:24.419715 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:48:24.419689    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab296ad3_b845_4bd9_a48c_492cd1469ddd.slice/crio-0f102c0948053071a5d48b509b0faffd4572756ecea081dcdda7d7d1dd3f58e7 WatchSource:0}: Error finding container 0f102c0948053071a5d48b509b0faffd4572756ecea081dcdda7d7d1dd3f58e7: Status 404 returned error can't find the container with id 0f102c0948053071a5d48b509b0faffd4572756ecea081dcdda7d7d1dd3f58e7
Oct 30 19:48:25.384890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:25.384850    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r" event={"ID":"ab296ad3-b845-4bd9-a48c-492cd1469ddd","Type":"ContainerStarted","Data":"ec475bc82e1b5a809016b79181425e7d9f24af0890789f773d2b6610f08f86ec"}
Oct 30 19:48:25.385364 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:25.384925    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r" event={"ID":"ab296ad3-b845-4bd9-a48c-492cd1469ddd","Type":"ContainerStarted","Data":"0f102c0948053071a5d48b509b0faffd4572756ecea081dcdda7d7d1dd3f58e7"}
Oct 30 19:48:26.399559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.399521    6777 generic.go:356] "Generic (PLEG): container finished" podID="ab296ad3-b845-4bd9-a48c-492cd1469ddd" containerID="ec475bc82e1b5a809016b79181425e7d9f24af0890789f773d2b6610f08f86ec" exitCode=0
Oct 30 19:48:26.400019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.399585    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r" event={"ID":"ab296ad3-b845-4bd9-a48c-492cd1469ddd","Type":"ContainerDied","Data":"ec475bc82e1b5a809016b79181425e7d9f24af0890789f773d2b6610f08f86ec"}
Oct 30 19:48:26.406979 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.406945    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main" containerID="cri-o://c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93" gracePeriod=45
Oct 30 19:48:26.699525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.699498    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:48:26.814995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.814957    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-home\") pod \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") "
Oct 30 19:48:26.815227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815086    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tls-certs\") pod \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") "
Oct 30 19:48:26.815227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815133    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-kube-api-access-lt96j\" (UniqueName: \"kubernetes.io/projected/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-kube-api-access-lt96j\") pod \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") "
Oct 30 19:48:26.815227 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815202    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tmp-dir\") pod \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") "
Oct 30 19:48:26.815378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815257    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-model-cache\") pod \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") "
Oct 30 19:48:26.815378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815343    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-dshm\") pod \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\" (UID: \"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd\") "
Oct 30 19:48:26.815378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815354    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-home" pod "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" (UID: "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:26.815582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815523    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tmp-dir" pod "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" (UID: "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:26.815679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815598    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-model-cache" pod "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" (UID: "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:26.815902 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815885    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:26.815986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815904    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:26.815986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.815913    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:26.817463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.817437    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-dshm" pod "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" (UID: "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:26.817553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.817479    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-kube-api-access-lt96j" pod "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" (UID: "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd"). InnerVolumeSpecName "kube-api-access-lt96j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:26.817553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.817494    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tls-certs" pod "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" (UID: "fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:26.917694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.917657    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:26.917694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.917688    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:26.917927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:26.917703    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt96j\" (UniqueName: \"kubernetes.io/projected/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd-kube-api-access-lt96j\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.408801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.408760    6777 generic.go:356] "Generic (PLEG): container finished" podID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerID="c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93" exitCode=0
Oct 30 19:48:27.408801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.408808    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"
Oct 30 19:48:27.409388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.408837    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" event={"ID":"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd","Type":"ContainerDied","Data":"c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93"}
Oct 30 19:48:27.409388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.408879    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f" event={"ID":"fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd","Type":"ContainerDied","Data":"20f21135b7db5558afd1acebc3b980d274356cc004c9f74ffb0f8816012e9f51"}
Oct 30 19:48:27.409388 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.408894    6777 scope.go:122] "RemoveContainer" containerID="c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93"
Oct 30 19:48:27.420852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.420825    6777 scope.go:122] "RemoveContainer" containerID="c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93"
Oct 30 19:48:27.421273 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:27.421250    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93\": container with ID starting with c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93 not found: ID does not exist" containerID="c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93"
Oct 30 19:48:27.421381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.421283    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93"} err="failed to get container status \"c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93\": rpc error: code = NotFound desc = could not find container \"c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93\": container with ID starting with c96a18315169bf4e9c9a84aa6c755241f407c94d5900257349b741ea55a39b93 not found: ID does not exist"
Oct 30 19:48:27.434975 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.434933    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"]
Oct 30 19:48:27.438903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.438872    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/premium-w4-27b5d5-kserve-b44b4c977-tlk5f"]
Oct 30 19:48:27.573471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.573438    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:27.654503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.654462    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" containerID="cri-o://7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0" gracePeriod=45
Oct 30 19:48:27.726748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.726711    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab296ad3-b845-4bd9-a48c-492cd1469ddd-kube-api-access-9c59n\" (UniqueName: \"kubernetes.io/projected/ab296ad3-b845-4bd9-a48c-492cd1469ddd-kube-api-access-9c59n\") pod \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") "
Oct 30 19:48:27.726934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.726760    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-xks-ca\") pod \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") "
Oct 30 19:48:27.726934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.726895    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-openshift-ca\") pod \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\" (UID: \"ab296ad3-b845-4bd9-a48c-492cd1469ddd\") "
Oct 30 19:48:27.727341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.727311    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-openshift-ca" pod "ab296ad3-b845-4bd9-a48c-492cd1469ddd" (UID: "ab296ad3-b845-4bd9-a48c-492cd1469ddd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:27.727467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.727405    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.728869 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.728844    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-xks-ca" pod "ab296ad3-b845-4bd9-a48c-492cd1469ddd" (UID: "ab296ad3-b845-4bd9-a48c-492cd1469ddd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:27.728951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.728921    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab296ad3-b845-4bd9-a48c-492cd1469ddd-kube-api-access-9c59n" pod "ab296ad3-b845-4bd9-a48c-492cd1469ddd" (UID: "ab296ad3-b845-4bd9-a48c-492cd1469ddd"). InnerVolumeSpecName "kube-api-access-9c59n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:27.828742 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.828709    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c59n\" (UniqueName: \"kubernetes.io/projected/ab296ad3-b845-4bd9-a48c-492cd1469ddd-kube-api-access-9c59n\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.828742 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.828743    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab296ad3-b845-4bd9-a48c-492cd1469ddd-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.922794 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:27.922768    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:48:28.031018 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.030967    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-dshm\") pod \"f64afb4d-4a06-410b-b249-d7fdeccb0226\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") "
Oct 30 19:48:28.031234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.031212    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f64afb4d-4a06-410b-b249-d7fdeccb0226-kube-api-access-8t6tw\" (UniqueName: \"kubernetes.io/projected/f64afb4d-4a06-410b-b249-d7fdeccb0226-kube-api-access-8t6tw\") pod \"f64afb4d-4a06-410b-b249-d7fdeccb0226\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") "
Oct 30 19:48:28.031339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.031321    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f64afb4d-4a06-410b-b249-d7fdeccb0226-tls-certs\" (UniqueName: \"kubernetes.io/secret/f64afb4d-4a06-410b-b249-d7fdeccb0226-tls-certs\") pod \"f64afb4d-4a06-410b-b249-d7fdeccb0226\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") "
Oct 30 19:48:28.031422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.031370    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-home\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-home\") pod \"f64afb4d-4a06-410b-b249-d7fdeccb0226\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") "
Oct 30 19:48:28.031422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.031410    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-tmp-dir\") pod \"f64afb4d-4a06-410b-b249-d7fdeccb0226\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") "
Oct 30 19:48:28.031562 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.031525    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-model-cache\") pod \"f64afb4d-4a06-410b-b249-d7fdeccb0226\" (UID: \"f64afb4d-4a06-410b-b249-d7fdeccb0226\") "
Oct 30 19:48:28.032153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.032016    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-tmp-dir" pod "f64afb4d-4a06-410b-b249-d7fdeccb0226" (UID: "f64afb4d-4a06-410b-b249-d7fdeccb0226"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.032153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.032048    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-model-cache" pod "f64afb4d-4a06-410b-b249-d7fdeccb0226" (UID: "f64afb4d-4a06-410b-b249-d7fdeccb0226"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.032153 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.032127    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-home" pod "f64afb4d-4a06-410b-b249-d7fdeccb0226" (UID: "f64afb4d-4a06-410b-b249-d7fdeccb0226"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.034214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.034186    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64afb4d-4a06-410b-b249-d7fdeccb0226-kube-api-access-8t6tw" pod "f64afb4d-4a06-410b-b249-d7fdeccb0226" (UID: "f64afb4d-4a06-410b-b249-d7fdeccb0226"). InnerVolumeSpecName "kube-api-access-8t6tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:28.035766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.035744    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-dshm" pod "f64afb4d-4a06-410b-b249-d7fdeccb0226" (UID: "f64afb4d-4a06-410b-b249-d7fdeccb0226"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.068282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.068239    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f64afb4d-4a06-410b-b249-d7fdeccb0226-tls-certs" pod "f64afb4d-4a06-410b-b249-d7fdeccb0226" (UID: "f64afb4d-4a06-410b-b249-d7fdeccb0226"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:28.113431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.113396    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" path="/var/lib/kubelet/pods/fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd/volumes"
Oct 30 19:48:28.133207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.133171    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6tw\" (UniqueName: \"kubernetes.io/projected/f64afb4d-4a06-410b-b249-d7fdeccb0226-kube-api-access-8t6tw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.133207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.133205    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64afb4d-4a06-410b-b249-d7fdeccb0226-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.133367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.133218    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.133367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.133230    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.133367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.133242    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.133367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.133256    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64afb4d-4a06-410b-b249-d7fdeccb0226-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.256135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.256014    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:48:28.479093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.472590    6777 generic.go:356] "Generic (PLEG): container finished" podID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerID="7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0" exitCode=0
Oct 30 19:48:28.479093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.472655    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"
Oct 30 19:48:28.479093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.472802    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" event={"ID":"f64afb4d-4a06-410b-b249-d7fdeccb0226","Type":"ContainerDied","Data":"7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0"}
Oct 30 19:48:28.479093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.472837    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" event={"ID":"f64afb4d-4a06-410b-b249-d7fdeccb0226","Type":"ContainerDied","Data":"4fc450843f2cd643595b96ba6fc07e9c8d7fe2f5a01a066d98ee5b36f034d51c"}
Oct 30 19:48:28.479093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.472854    6777 scope.go:122] "RemoveContainer" containerID="7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0"
Oct 30 19:48:28.479093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.475597    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r" event={"ID":"ab296ad3-b845-4bd9-a48c-492cd1469ddd","Type":"ContainerDied","Data":"0f102c0948053071a5d48b509b0faffd4572756ecea081dcdda7d7d1dd3f58e7"}
Oct 30 19:48:28.479093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.475715    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"
Oct 30 19:48:28.479093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.477449    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f102c0948053071a5d48b509b0faffd4572756ecea081dcdda7d7d1dd3f58e7"
Oct 30 19:48:28.486907 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.486833    6777 scope.go:122] "RemoveContainer" containerID="7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0"
Oct 30 19:48:28.487304 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:28.487274    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0\": container with ID starting with 7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0 not found: ID does not exist" containerID="7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0"
Oct 30 19:48:28.487410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.487314    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0"} err="failed to get container status \"7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0\": rpc error: code = NotFound desc = could not find container \"7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0\": container with ID starting with 7b1b60c4e06b15f802a1b0f87ef2d51d0e5e3c83f2ee5c30c33aa641fb8498f0 not found: ID does not exist"
Oct 30 19:48:28.502235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.502199    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"]
Oct 30 19:48:28.509179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.509098    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4"]
Oct 30 19:48:28.837980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:28.837931    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/distinct-2-w5-2d5d53-kserve-84779bddf6-5jvn4" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:29.033989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.033960    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:48:29.209882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.209788    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcdde26a-de3b-4a41-812a-cefa04196839-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdde26a-de3b-4a41-812a-cefa04196839-istio-podinfo\") pod \"fcdde26a-de3b-4a41-812a-cefa04196839\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") "
Oct 30 19:48:29.209882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.209845    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-certs\") pod \"fcdde26a-de3b-4a41-812a-cefa04196839\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") "
Oct 30 19:48:29.210176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.209889    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-kube-api-access-f46xz\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-kube-api-access-f46xz\") pod \"fcdde26a-de3b-4a41-812a-cefa04196839\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") "
Oct 30 19:48:29.210176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.209982    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-socket\") pod \"fcdde26a-de3b-4a41-812a-cefa04196839\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") "
Oct 30 19:48:29.210176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210099    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-credential-socket\") pod \"fcdde26a-de3b-4a41-812a-cefa04196839\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") "
Oct 30 19:48:29.210176 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210166    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-envoy\") pod \"fcdde26a-de3b-4a41-812a-cefa04196839\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") "
Oct 30 19:48:29.210404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210215    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-certs" pod "fcdde26a-de3b-4a41-812a-cefa04196839" (UID: "fcdde26a-de3b-4a41-812a-cefa04196839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:29.210404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210294    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-socket" pod "fcdde26a-de3b-4a41-812a-cefa04196839" (UID: "fcdde26a-de3b-4a41-812a-cefa04196839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:29.210404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210310    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-data\") pod \"fcdde26a-de3b-4a41-812a-cefa04196839\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") "
Oct 30 19:48:29.210404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210308    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-credential-socket" pod "fcdde26a-de3b-4a41-812a-cefa04196839" (UID: "fcdde26a-de3b-4a41-812a-cefa04196839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:29.210404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210388    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-istio-token\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-istio-token\") pod \"fcdde26a-de3b-4a41-812a-cefa04196839\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") "
Oct 30 19:48:29.210645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210487    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdde26a-de3b-4a41-812a-cefa04196839-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdde26a-de3b-4a41-812a-cefa04196839-istiod-ca-cert\") pod \"fcdde26a-de3b-4a41-812a-cefa04196839\" (UID: \"fcdde26a-de3b-4a41-812a-cefa04196839\") "
Oct 30 19:48:29.210645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210513    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-data" pod "fcdde26a-de3b-4a41-812a-cefa04196839" (UID: "fcdde26a-de3b-4a41-812a-cefa04196839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:29.210929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.210905    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdde26a-de3b-4a41-812a-cefa04196839-istiod-ca-cert" pod "fcdde26a-de3b-4a41-812a-cefa04196839" (UID: "fcdde26a-de3b-4a41-812a-cefa04196839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:29.211103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.211052    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.211146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.211113    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.211146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.211128    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.211146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.211140    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.211245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.211153    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdde26a-de3b-4a41-812a-cefa04196839-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.212335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.212305    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-envoy" pod "fcdde26a-de3b-4a41-812a-cefa04196839" (UID: "fcdde26a-de3b-4a41-812a-cefa04196839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:29.212428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.212345    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-kube-api-access-f46xz" pod "fcdde26a-de3b-4a41-812a-cefa04196839" (UID: "fcdde26a-de3b-4a41-812a-cefa04196839"). InnerVolumeSpecName "kube-api-access-f46xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:29.212495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.212475    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcdde26a-de3b-4a41-812a-cefa04196839-istio-podinfo" pod "fcdde26a-de3b-4a41-812a-cefa04196839" (UID: "fcdde26a-de3b-4a41-812a-cefa04196839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:29.212587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.212571    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-istio-token" pod "fcdde26a-de3b-4a41-812a-cefa04196839" (UID: "fcdde26a-de3b-4a41-812a-cefa04196839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:29.312425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.312386    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.312425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.312420    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdde26a-de3b-4a41-812a-cefa04196839-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.312425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.312430    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f46xz\" (UniqueName: \"kubernetes.io/projected/fcdde26a-de3b-4a41-812a-cefa04196839-kube-api-access-f46xz\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.312677 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.312439    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdde26a-de3b-4a41-812a-cefa04196839-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.485097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.484992    6777 generic.go:356] "Generic (PLEG): container finished" podID="fcdde26a-de3b-4a41-812a-cefa04196839" containerID="50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a" exitCode=0
Oct 30 19:48:29.485097 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.485041    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"
Oct 30 19:48:29.485650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.485140    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" event={"ID":"fcdde26a-de3b-4a41-812a-cefa04196839","Type":"ContainerDied","Data":"50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a"}
Oct 30 19:48:29.485650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.485198    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds" event={"ID":"fcdde26a-de3b-4a41-812a-cefa04196839","Type":"ContainerDied","Data":"108c7a46e4c16b799827a7063df5cdcc5a20a8665c7702897421fb1345bcd70b"}
Oct 30 19:48:29.485650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.485230    6777 scope.go:122] "RemoveContainer" containerID="50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a"
Oct 30 19:48:29.497026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.497002    6777 scope.go:122] "RemoveContainer" containerID="50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a"
Oct 30 19:48:29.497361 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:29.497342    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a\": container with ID starting with 50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a not found: ID does not exist" containerID="50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a"
Oct 30 19:48:29.497412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.497371    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a"} err="failed to get container status \"50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a\": rpc error: code = NotFound desc = could not find container \"50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a\": container with ID starting with 50fd2b9d35c5beb528d642589e3c53c3afa3f702e2147dd33ed84b39b2cce56a not found: ID does not exist"
Oct 30 19:48:29.510536 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.510500    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"]
Oct 30 19:48:29.515012 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:29.514977    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-77zds"]
Oct 30 19:48:30.122850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:30.122810    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" path="/var/lib/kubelet/pods/f64afb4d-4a06-410b-b249-d7fdeccb0226/volumes"
Oct 30 19:48:30.123784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:30.123764    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" path="/var/lib/kubelet/pods/fcdde26a-de3b-4a41-812a-cefa04196839/volumes"
Oct 30 19:48:33.898429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:33.898392    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"]
Oct 30 19:48:33.906142 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:33.906103    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"]
Oct 30 19:48:36.665586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:36.665552    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"]
Oct 30 19:48:36.666030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:36.665812    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy" containerID="cri-o://dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d" gracePeriod=30
Oct 30 19:48:36.668931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:36.668896    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"]
Oct 30 19:48:36.669277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:36.669211    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy" containerID="cri-o://0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859" gracePeriod=30
Oct 30 19:48:36.672151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:36.672122    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"]
Oct 30 19:48:36.672419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:36.672371    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy" containerID="cri-o://551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7" gracePeriod=30
Oct 30 19:48:38.267148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:38.267108    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"]
Oct 30 19:48:38.270482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:38.270441    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8f3ff4-9xz2r"]
Oct 30 19:48:38.898858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:38.898817    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp" podUID="f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" containerName="payload-processing" containerID="cri-o://16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f" gracePeriod=25
Oct 30 19:48:38.906473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:38.906437    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx" podUID="b9dcef83-3bad-4f42-81a8-3051ec762234" containerName="payload-pre-processing" containerID="cri-o://c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81" gracePeriod=25
Oct 30 19:48:40.112290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:40.112251    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab296ad3-b845-4bd9-a48c-492cd1469ddd" path="/var/lib/kubelet/pods/ab296ad3-b845-4bd9-a48c-492cd1469ddd/volumes"
Oct 30 19:48:40.946804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:40.946763    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:40.946996 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:40.946826    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:41.840941 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:41.840619    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx" podUID="b9dcef83-3bad-4f42-81a8-3051ec762234" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:41.843313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:41.843240    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp" podUID="f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:41.973109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:41.966349    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:48:42.018535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.018201    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:48:42.045617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.045590    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:48:42.088089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088018    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-data\") pod \"0b3631e6-8e99-43f2-8e80-44524d73d34a\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") "
Oct 30 19:48:42.088289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088125    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-token\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-token\") pod \"01352da7-5b23-44ec-addb-79a6f4a364b3\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") "
Oct 30 19:48:42.088289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088168    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-credential-socket\") pod \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") "
Oct 30 19:48:42.088289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088201    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-podinfo\") pod \"0b3631e6-8e99-43f2-8e80-44524d73d34a\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") "
Oct 30 19:48:42.088289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088230    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b3631e6-8e99-43f2-8e80-44524d73d34a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3631e6-8e99-43f2-8e80-44524d73d34a-istiod-ca-cert\") pod \"0b3631e6-8e99-43f2-8e80-44524d73d34a\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") "
Oct 30 19:48:42.088289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088257    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-credential-socket\") pod \"01352da7-5b23-44ec-addb-79a6f4a364b3\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") "
Oct 30 19:48:42.088485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088306    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-kube-api-access-4mpst\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-kube-api-access-4mpst\") pod \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") "
Oct 30 19:48:42.088485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088361    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-podinfo\") pod \"01352da7-5b23-44ec-addb-79a6f4a364b3\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") "
Oct 30 19:48:42.088485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088426    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-certs\") pod \"0b3631e6-8e99-43f2-8e80-44524d73d34a\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") "
Oct 30 19:48:42.088485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088455    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-envoy\") pod \"01352da7-5b23-44ec-addb-79a6f4a364b3\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") "
Oct 30 19:48:42.088626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088506    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-envoy\") pod \"0b3631e6-8e99-43f2-8e80-44524d73d34a\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") "
Oct 30 19:48:42.088626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088536    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-data\") pod \"01352da7-5b23-44ec-addb-79a6f4a364b3\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") "
Oct 30 19:48:42.088626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088566    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-certs\") pod \"01352da7-5b23-44ec-addb-79a6f4a364b3\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") "
Oct 30 19:48:42.088626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088607    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-data\") pod \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") "
Oct 30 19:48:42.088752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088674    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-socket\") pod \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") "
Oct 30 19:48:42.088752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088713    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-kube-api-access-td49m\") pod \"0b3631e6-8e99-43f2-8e80-44524d73d34a\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") "
Oct 30 19:48:42.088821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088759    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-socket\") pod \"0b3631e6-8e99-43f2-8e80-44524d73d34a\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") "
Oct 30 19:48:42.088821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088791    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-socket\") pod \"01352da7-5b23-44ec-addb-79a6f4a364b3\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") "
Oct 30 19:48:42.088888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088828    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-podinfo\") pod \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") "
Oct 30 19:48:42.088888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088860    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-credential-socket\") pod \"0b3631e6-8e99-43f2-8e80-44524d73d34a\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") "
Oct 30 19:48:42.088963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088896    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-certs\") pod \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") "
Oct 30 19:48:42.088998 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.088983    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istiod-ca-cert\") pod \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") "
Oct 30 19:48:42.089085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.089048    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-envoy\") pod \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") "
Oct 30 19:48:42.089129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.089105    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-token\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-token\") pod \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\" (UID: \"05bc02d6-1cd9-44a3-9cb8-97cf81560051\") "
Oct 30 19:48:42.089190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.089177    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01352da7-5b23-44ec-addb-79a6f4a364b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01352da7-5b23-44ec-addb-79a6f4a364b3-istiod-ca-cert\") pod \"01352da7-5b23-44ec-addb-79a6f4a364b3\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") "
Oct 30 19:48:42.089229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.089210    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-kube-api-access-xvmkk\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-kube-api-access-xvmkk\") pod \"01352da7-5b23-44ec-addb-79a6f4a364b3\" (UID: \"01352da7-5b23-44ec-addb-79a6f4a364b3\") "
Oct 30 19:48:42.089282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.089270    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-token\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-token\") pod \"0b3631e6-8e99-43f2-8e80-44524d73d34a\" (UID: \"0b3631e6-8e99-43f2-8e80-44524d73d34a\") "
Oct 30 19:48:42.090967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.090739    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-credential-socket" pod "0b3631e6-8e99-43f2-8e80-44524d73d34a" (UID: "0b3631e6-8e99-43f2-8e80-44524d73d34a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.090967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.090748    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-certs" pod "05bc02d6-1cd9-44a3-9cb8-97cf81560051" (UID: "05bc02d6-1cd9-44a3-9cb8-97cf81560051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.091419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.091356    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istiod-ca-cert" pod "05bc02d6-1cd9-44a3-9cb8-97cf81560051" (UID: "05bc02d6-1cd9-44a3-9cb8-97cf81560051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:42.092801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.092432    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-token" pod "0b3631e6-8e99-43f2-8e80-44524d73d34a" (UID: "0b3631e6-8e99-43f2-8e80-44524d73d34a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:42.092891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.092839    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01352da7-5b23-44ec-addb-79a6f4a364b3-istiod-ca-cert" pod "01352da7-5b23-44ec-addb-79a6f4a364b3" (UID: "01352da7-5b23-44ec-addb-79a6f4a364b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:42.093105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.093046    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-kube-api-access-td49m" pod "0b3631e6-8e99-43f2-8e80-44524d73d34a" (UID: "0b3631e6-8e99-43f2-8e80-44524d73d34a"). InnerVolumeSpecName "kube-api-access-td49m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:42.093225 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.093179    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-credential-socket" pod "01352da7-5b23-44ec-addb-79a6f4a364b3" (UID: "01352da7-5b23-44ec-addb-79a6f4a364b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.093402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.093288    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-data" pod "05bc02d6-1cd9-44a3-9cb8-97cf81560051" (UID: "05bc02d6-1cd9-44a3-9cb8-97cf81560051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.093478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.093396    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-data" pod "0b3631e6-8e99-43f2-8e80-44524d73d34a" (UID: "0b3631e6-8e99-43f2-8e80-44524d73d34a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.093545 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.093490    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-socket" pod "05bc02d6-1cd9-44a3-9cb8-97cf81560051" (UID: "05bc02d6-1cd9-44a3-9cb8-97cf81560051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.094144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.094119    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-socket" pod "01352da7-5b23-44ec-addb-79a6f4a364b3" (UID: "01352da7-5b23-44ec-addb-79a6f4a364b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.094524 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.094503    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-socket" pod "0b3631e6-8e99-43f2-8e80-44524d73d34a" (UID: "0b3631e6-8e99-43f2-8e80-44524d73d34a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.094874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.094844    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-token" pod "05bc02d6-1cd9-44a3-9cb8-97cf81560051" (UID: "05bc02d6-1cd9-44a3-9cb8-97cf81560051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:42.095094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.095045    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3631e6-8e99-43f2-8e80-44524d73d34a-istiod-ca-cert" pod "0b3631e6-8e99-43f2-8e80-44524d73d34a" (UID: "0b3631e6-8e99-43f2-8e80-44524d73d34a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:42.095799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.095668    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-certs" pod "0b3631e6-8e99-43f2-8e80-44524d73d34a" (UID: "0b3631e6-8e99-43f2-8e80-44524d73d34a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.095901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.095839    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-credential-socket" pod "05bc02d6-1cd9-44a3-9cb8-97cf81560051" (UID: "05bc02d6-1cd9-44a3-9cb8-97cf81560051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.095968 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.095922    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-data" pod "01352da7-5b23-44ec-addb-79a6f4a364b3" (UID: "01352da7-5b23-44ec-addb-79a6f4a364b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.096190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.096164    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-token" pod "01352da7-5b23-44ec-addb-79a6f4a364b3" (UID: "01352da7-5b23-44ec-addb-79a6f4a364b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:42.096708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.096674    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-podinfo" pod "0b3631e6-8e99-43f2-8e80-44524d73d34a" (UID: "0b3631e6-8e99-43f2-8e80-44524d73d34a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:42.096862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.096814    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-kube-api-access-4mpst" pod "05bc02d6-1cd9-44a3-9cb8-97cf81560051" (UID: "05bc02d6-1cd9-44a3-9cb8-97cf81560051"). InnerVolumeSpecName "kube-api-access-4mpst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:42.097206 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.097185    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-envoy" pod "01352da7-5b23-44ec-addb-79a6f4a364b3" (UID: "01352da7-5b23-44ec-addb-79a6f4a364b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.097834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.097810    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-kube-api-access-xvmkk" pod "01352da7-5b23-44ec-addb-79a6f4a364b3" (UID: "01352da7-5b23-44ec-addb-79a6f4a364b3"). InnerVolumeSpecName "kube-api-access-xvmkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:42.097911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.097881    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-certs" pod "01352da7-5b23-44ec-addb-79a6f4a364b3" (UID: "01352da7-5b23-44ec-addb-79a6f4a364b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.097990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.097970    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-envoy" pod "0b3631e6-8e99-43f2-8e80-44524d73d34a" (UID: "0b3631e6-8e99-43f2-8e80-44524d73d34a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.098387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.098363    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-envoy" pod "05bc02d6-1cd9-44a3-9cb8-97cf81560051" (UID: "05bc02d6-1cd9-44a3-9cb8-97cf81560051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:42.099132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.099106    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-podinfo" pod "01352da7-5b23-44ec-addb-79a6f4a364b3" (UID: "01352da7-5b23-44ec-addb-79a6f4a364b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:42.099628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.099610    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-podinfo" pod "05bc02d6-1cd9-44a3-9cb8-97cf81560051" (UID: "05bc02d6-1cd9-44a3-9cb8-97cf81560051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:42.173422 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.173385    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"]
Oct 30 19:48:42.182112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.182073    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"]
Oct 30 19:48:42.191853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191754    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.191853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191791    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.191853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191808    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.191853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191822    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.191853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191834    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.191853 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191848    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191864    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01352da7-5b23-44ec-addb-79a6f4a364b3-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191880    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvmkk\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-kube-api-access-xvmkk\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191897    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191910    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191924    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191937    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191950    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191963    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3631e6-8e99-43f2-8e80-44524d73d34a-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191976    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.191989    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpst\" (UniqueName: \"kubernetes.io/projected/05bc02d6-1cd9-44a3-9cb8-97cf81560051-kube-api-access-4mpst\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192005    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192018    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192031    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192047    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192098    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192112    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192126    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192139    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bc02d6-1cd9-44a3-9cb8-97cf81560051-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192153    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/0b3631e6-8e99-43f2-8e80-44524d73d34a-kube-api-access-td49m\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192167    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3631e6-8e99-43f2-8e80-44524d73d34a-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.192315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.192182    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01352da7-5b23-44ec-addb-79a6f4a364b3-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.576189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.576153    6777 generic.go:356] "Generic (PLEG): container finished" podID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerID="0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859" exitCode=0
Oct 30 19:48:42.576366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.576205    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"
Oct 30 19:48:42.576366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.576243    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" event={"ID":"05bc02d6-1cd9-44a3-9cb8-97cf81560051","Type":"ContainerDied","Data":"0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859"}
Oct 30 19:48:42.576366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.576296    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l" event={"ID":"05bc02d6-1cd9-44a3-9cb8-97cf81560051","Type":"ContainerDied","Data":"90ca5ab9ebd2bffa6f2b2b1fb2e44c5fb26525b2079faff67a349af28d867a14"}
Oct 30 19:48:42.576366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.576314    6777 scope.go:122] "RemoveContainer" containerID="0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859"
Oct 30 19:48:42.577729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.577708    6777 generic.go:356] "Generic (PLEG): container finished" podID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerID="dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d" exitCode=0
Oct 30 19:48:42.577843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.577761    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"
Oct 30 19:48:42.577843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.577769    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" event={"ID":"01352da7-5b23-44ec-addb-79a6f4a364b3","Type":"ContainerDied","Data":"dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d"}
Oct 30 19:48:42.577843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.577824    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269" event={"ID":"01352da7-5b23-44ec-addb-79a6f4a364b3","Type":"ContainerDied","Data":"b6b33ba481ce8c59f31139363d5cf181909a6f97ebc6abf41ffe568bf51f3cc0"}
Oct 30 19:48:42.579730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.579703    6777 generic.go:356] "Generic (PLEG): container finished" podID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerID="551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7" exitCode=0
Oct 30 19:48:42.579843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.579741    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"
Oct 30 19:48:42.579843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.579747    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" event={"ID":"0b3631e6-8e99-43f2-8e80-44524d73d34a","Type":"ContainerDied","Data":"551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7"}
Oct 30 19:48:42.579843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.579793    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59" event={"ID":"0b3631e6-8e99-43f2-8e80-44524d73d34a","Type":"ContainerDied","Data":"117a40f24ef0f9b40251407d31c35b62b37c1a98c5318ba5a5255898b8cb56ea"}
Oct 30 19:48:42.591632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.591608    6777 scope.go:122] "RemoveContainer" containerID="0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859"
Oct 30 19:48:42.592004 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:42.591978    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859\": container with ID starting with 0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859 not found: ID does not exist" containerID="0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859"
Oct 30 19:48:42.592086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.592018    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859"} err="failed to get container status \"0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859\": rpc error: code = NotFound desc = could not find container \"0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859\": container with ID starting with 0e08ad1434e6b9628202b6133ec3064e9cc1b785032dd3faba332af1b0434859 not found: ID does not exist"
Oct 30 19:48:42.592086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.592041    6777 scope.go:122] "RemoveContainer" containerID="dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d"
Oct 30 19:48:42.603222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.603185    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"]
Oct 30 19:48:42.604069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.604041    6777 scope.go:122] "RemoveContainer" containerID="dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d"
Oct 30 19:48:42.604461 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:42.604439    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d\": container with ID starting with dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d not found: ID does not exist" containerID="dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d"
Oct 30 19:48:42.604529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.604468    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d"} err="failed to get container status \"dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d\": rpc error: code = NotFound desc = could not find container \"dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d\": container with ID starting with dead655e71111395a85258400304348b60ef4bb2235249e60ca190cd8024ad2d not found: ID does not exist"
Oct 30 19:48:42.604529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.604490    6777 scope.go:122] "RemoveContainer" containerID="551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7"
Oct 30 19:48:42.615112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.615027    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-v7r7l"]
Oct 30 19:48:42.616884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.616861    6777 scope.go:122] "RemoveContainer" containerID="551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7"
Oct 30 19:48:42.617265 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:42.617245    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7\": container with ID starting with 551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7 not found: ID does not exist" containerID="551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7"
Oct 30 19:48:42.617328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.617273    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7"} err="failed to get container status \"551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7\": rpc error: code = NotFound desc = could not find container \"551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7\": container with ID starting with 551cdcfb2b306bb37de04028047f9cee8131f7c7d75ed21e6a7fcd37f8ba04c7 not found: ID does not exist"
Oct 30 19:48:42.638124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.638074    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"]
Oct 30 19:48:42.645727 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.645692    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-7dj59"]
Oct 30 19:48:42.659257 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.659224    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"]
Oct 30 19:48:42.668746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:42.668713    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-gz269"]
Oct 30 19:48:43.648333 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:43.648270    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:48:43.796897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:43.796858    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"]
Oct 30 19:48:43.797175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:43.797146    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy" containerID="cri-o://0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c" gracePeriod=30
Oct 30 19:48:43.916013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:43.915922    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"]
Oct 30 19:48:43.916357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:43.916180    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy" containerID="cri-o://cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9" gracePeriod=30
Oct 30 19:48:44.139889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:44.139849    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" path="/var/lib/kubelet/pods/01352da7-5b23-44ec-addb-79a6f4a364b3/volumes"
Oct 30 19:48:44.140644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:44.140617    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" path="/var/lib/kubelet/pods/05bc02d6-1cd9-44a3-9cb8-97cf81560051/volumes"
Oct 30 19:48:44.141301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:44.141277    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" path="/var/lib/kubelet/pods/0b3631e6-8e99-43f2-8e80-44524d73d34a/volumes"
Oct 30 19:48:47.125711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.125667    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"]
Oct 30 19:48:47.126430 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126408    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ab296ad3-b845-4bd9-a48c-492cd1469ddd" containerName="revoke-keys"
Oct 30 19:48:47.126533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126432    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab296ad3-b845-4bd9-a48c-492cd1469ddd" containerName="revoke-keys"
Oct 30 19:48:47.126533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126462    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main"
Oct 30 19:48:47.126533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126470    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main"
Oct 30 19:48:47.126533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126487    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy"
Oct 30 19:48:47.126533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126495    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy"
Oct 30 19:48:47.126533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126505    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main"
Oct 30 19:48:47.126533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126512    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main"
Oct 30 19:48:47.126533 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126530    6777 cpu_manager.go:414] "containerMap: removing container" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126537    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126553    6777 cpu_manager.go:414] "containerMap: removing container" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126560    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126576    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126584    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126735    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6fc38a-2eaf-4d7b-8f86-1a983d77fcbd" containerName="main"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126752    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="01352da7-5b23-44ec-addb-79a6f4a364b3" containerName="istio-proxy"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126763    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="05bc02d6-1cd9-44a3-9cb8-97cf81560051" containerName="istio-proxy"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126773    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdde26a-de3b-4a41-812a-cefa04196839" containerName="istio-proxy"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126788    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab296ad3-b845-4bd9-a48c-492cd1469ddd" containerName="revoke-keys"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126800    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f64afb4d-4a06-410b-b249-d7fdeccb0226" containerName="main"
Oct 30 19:48:47.126939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.126813    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b3631e6-8e99-43f2-8e80-44524d73d34a" containerName="istio-proxy"
Oct 30 19:48:47.131921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.131896    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.136505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.136480    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:47.138576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.138547    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"]
Oct 30 19:48:47.138887 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.138862    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:47.139015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.138890    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:47.139015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.138891    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:47.174340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.174294    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp" podUID="e67fd308-ff22-476f-82a1-e22d41c8381e" containerName="payload-processing" containerID="cri-o://32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8" gracePeriod=25
Oct 30 19:48:47.183218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.183177    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr" podUID="095c2e53-0ad0-40f6-af86-0bad69490983" containerName="payload-pre-processing" containerID="cri-o://976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58" gracePeriod=25
Oct 30 19:48:47.269327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.269287    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.269531 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.269346    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.269621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.269594    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjfw\" (UniqueName: \"kubernetes.io/projected/f54a1f71-c71b-402c-b7a5-14859066df28-kube-api-access-qjjfw\") pod \"maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.371391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.371345    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.371598 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.371449    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.371699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.371639    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjfw\" (UniqueName: \"kubernetes.io/projected/f54a1f71-c71b-402c-b7a5-14859066df28-kube-api-access-qjjfw\") pod \"maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.372260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.372235    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.374500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.374475    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.386341 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.386271    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjfw\" (UniqueName: \"kubernetes.io/projected/f54a1f71-c71b-402c-b7a5-14859066df28-kube-api-access-qjjfw\") pod \"maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.450090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.450034    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:47.730544 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:47.730513    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"]
Oct 30 19:48:47.732295 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:48:47.732264    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54a1f71_c71b_402c_b7a5_14859066df28.slice/crio-cf34f83ede7e54b44b68a45c6c02c4604315093b43e6a48345890d12fbf11a10 WatchSource:0}: Error finding container cf34f83ede7e54b44b68a45c6c02c4604315093b43e6a48345890d12fbf11a10: Status 404 returned error can't find the container with id cf34f83ede7e54b44b68a45c6c02c4604315093b43e6a48345890d12fbf11a10
Oct 30 19:48:48.628590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:48.628553    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f" event={"ID":"f54a1f71-c71b-402c-b7a5-14859066df28","Type":"ContainerStarted","Data":"884decb53796c3f4c0a50218e289e2ce5988860ea2a1d379e35c256064d9009d"}
Oct 30 19:48:48.629005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:48.628613    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f" event={"ID":"f54a1f71-c71b-402c-b7a5-14859066df28","Type":"ContainerStarted","Data":"cf34f83ede7e54b44b68a45c6c02c4604315093b43e6a48345890d12fbf11a10"}
Oct 30 19:48:48.667087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:48.648770    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f" podStartSLOduration=1.64874887 podStartE2EDuration="1.64874887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:48.647872762 +0000 UTC m=+2510.644788147" watchObservedRunningTime="2026-10-30 19:48:48.64874887 +0000 UTC m=+2510.645664266"
Oct 30 19:48:49.058829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.058801    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.194537    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-data\") pod \"b7ded767-98c6-459e-9209-3c58dc4751c8\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") "
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.198623    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-kube-api-access-hqhm8\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-kube-api-access-hqhm8\") pod \"b7ded767-98c6-459e-9209-3c58dc4751c8\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") "
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.198706    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-socket\") pod \"b7ded767-98c6-459e-9209-3c58dc4751c8\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") "
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.198736    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ded767-98c6-459e-9209-3c58dc4751c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ded767-98c6-459e-9209-3c58dc4751c8-istiod-ca-cert\") pod \"b7ded767-98c6-459e-9209-3c58dc4751c8\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") "
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.198764    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-data" pod "b7ded767-98c6-459e-9209-3c58dc4751c8" (UID: "b7ded767-98c6-459e-9209-3c58dc4751c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.198792    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-token\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-token\") pod \"b7ded767-98c6-459e-9209-3c58dc4751c8\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") "
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.198891    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-podinfo\") pod \"b7ded767-98c6-459e-9209-3c58dc4751c8\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") "
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.198925    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-credential-socket\") pod \"b7ded767-98c6-459e-9209-3c58dc4751c8\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") "
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.198968    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-certs\") pod \"b7ded767-98c6-459e-9209-3c58dc4751c8\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") "
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.198999    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-envoy\") pod \"b7ded767-98c6-459e-9209-3c58dc4751c8\" (UID: \"b7ded767-98c6-459e-9209-3c58dc4751c8\") "
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.199163    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ded767-98c6-459e-9209-3c58dc4751c8-istiod-ca-cert" pod "b7ded767-98c6-459e-9209-3c58dc4751c8" (UID: "b7ded767-98c6-459e-9209-3c58dc4751c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.199408    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-credential-socket" pod "b7ded767-98c6-459e-9209-3c58dc4751c8" (UID: "b7ded767-98c6-459e-9209-3c58dc4751c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.199792    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-certs" pod "b7ded767-98c6-459e-9209-3c58dc4751c8" (UID: "b7ded767-98c6-459e-9209-3c58dc4751c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.199963    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-socket" pod "b7ded767-98c6-459e-9209-3c58dc4751c8" (UID: "b7ded767-98c6-459e-9209-3c58dc4751c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.200367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.200376    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.201277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.200399    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.201277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.200416    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.201277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.200429    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.201277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.200444    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ded767-98c6-459e-9209-3c58dc4751c8-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.202331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.202302    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-envoy" pod "b7ded767-98c6-459e-9209-3c58dc4751c8" (UID: "b7ded767-98c6-459e-9209-3c58dc4751c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.202331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.202309    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-podinfo" pod "b7ded767-98c6-459e-9209-3c58dc4751c8" (UID: "b7ded767-98c6-459e-9209-3c58dc4751c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:49.202541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.202490    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-token" pod "b7ded767-98c6-459e-9209-3c58dc4751c8" (UID: "b7ded767-98c6-459e-9209-3c58dc4751c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:49.202755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.202722    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-kube-api-access-hqhm8" pod "b7ded767-98c6-459e-9209-3c58dc4751c8" (UID: "b7ded767-98c6-459e-9209-3c58dc4751c8"). InnerVolumeSpecName "kube-api-access-hqhm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:49.213777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.213747    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:48:49.301709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.301665    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-socket\") pod \"96ca458c-b3a8-4c14-93b5-4da753fbc136\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") "
Oct 30 19:48:49.301918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.301814    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-podinfo\") pod \"96ca458c-b3a8-4c14-93b5-4da753fbc136\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") "
Oct 30 19:48:49.301918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.301853    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-kube-api-access-zbkt4\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-kube-api-access-zbkt4\") pod \"96ca458c-b3a8-4c14-93b5-4da753fbc136\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") "
Oct 30 19:48:49.301918 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.301900    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-certs\") pod \"96ca458c-b3a8-4c14-93b5-4da753fbc136\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") "
Oct 30 19:48:49.302139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.301942    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ca458c-b3a8-4c14-93b5-4da753fbc136-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ca458c-b3a8-4c14-93b5-4da753fbc136-istiod-ca-cert\") pod \"96ca458c-b3a8-4c14-93b5-4da753fbc136\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") "
Oct 30 19:48:49.302139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.302001    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-token\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-token\") pod \"96ca458c-b3a8-4c14-93b5-4da753fbc136\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") "
Oct 30 19:48:49.302139 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.302095    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-credential-socket\") pod \"96ca458c-b3a8-4c14-93b5-4da753fbc136\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") "
Oct 30 19:48:49.302302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.302181    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-data\") pod \"96ca458c-b3a8-4c14-93b5-4da753fbc136\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") "
Oct 30 19:48:49.302302 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.302277    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-envoy\") pod \"96ca458c-b3a8-4c14-93b5-4da753fbc136\" (UID: \"96ca458c-b3a8-4c14-93b5-4da753fbc136\") "
Oct 30 19:48:49.302403 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.302333    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-socket" pod "96ca458c-b3a8-4c14-93b5-4da753fbc136" (UID: "96ca458c-b3a8-4c14-93b5-4da753fbc136"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.302555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.302527    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-credential-socket" pod "96ca458c-b3a8-4c14-93b5-4da753fbc136" (UID: "96ca458c-b3a8-4c14-93b5-4da753fbc136"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.302634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.302582    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ca458c-b3a8-4c14-93b5-4da753fbc136-istiod-ca-cert" pod "96ca458c-b3a8-4c14-93b5-4da753fbc136" (UID: "96ca458c-b3a8-4c14-93b5-4da753fbc136"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:49.302957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.302750    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-certs" pod "96ca458c-b3a8-4c14-93b5-4da753fbc136" (UID: "96ca458c-b3a8-4c14-93b5-4da753fbc136"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.302957 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.302815    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-data" pod "96ca458c-b3a8-4c14-93b5-4da753fbc136" (UID: "96ca458c-b3a8-4c14-93b5-4da753fbc136"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.303182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.303108    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqhm8\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-kube-api-access-hqhm8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.303182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.303132    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.303182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.303148    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.303182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.303163    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.303182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.303176    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ded767-98c6-459e-9209-3c58dc4751c8-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.303380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.303190    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.303380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.303203    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ca458c-b3a8-4c14-93b5-4da753fbc136-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.303380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.303218    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.303380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.303231    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.304488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.304468    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-podinfo" pod "96ca458c-b3a8-4c14-93b5-4da753fbc136" (UID: "96ca458c-b3a8-4c14-93b5-4da753fbc136"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:49.304573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.304517    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-token" pod "96ca458c-b3a8-4c14-93b5-4da753fbc136" (UID: "96ca458c-b3a8-4c14-93b5-4da753fbc136"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:49.304819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.304794    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-envoy" pod "96ca458c-b3a8-4c14-93b5-4da753fbc136" (UID: "96ca458c-b3a8-4c14-93b5-4da753fbc136"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.304871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.304825    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-kube-api-access-zbkt4" pod "96ca458c-b3a8-4c14-93b5-4da753fbc136" (UID: "96ca458c-b3a8-4c14-93b5-4da753fbc136"). InnerVolumeSpecName "kube-api-access-zbkt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:49.404465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.404429    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.404465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.404461    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbkt4\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-kube-api-access-zbkt4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.404465 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.404474    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.404719 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.404487    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ca458c-b3a8-4c14-93b5-4da753fbc136-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.636222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.636187    6777 generic.go:356] "Generic (PLEG): container finished" podID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerID="cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9" exitCode=0
Oct 30 19:48:49.637169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.636241    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"
Oct 30 19:48:49.637169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.636272    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" event={"ID":"96ca458c-b3a8-4c14-93b5-4da753fbc136","Type":"ContainerDied","Data":"cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9"}
Oct 30 19:48:49.637169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.636329    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb" event={"ID":"96ca458c-b3a8-4c14-93b5-4da753fbc136","Type":"ContainerDied","Data":"8b3f806141d369c0dcd4bc7be97f2272893a2514c89b20417e19f6fed5fb7a71"}
Oct 30 19:48:49.637169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.636353    6777 scope.go:122] "RemoveContainer" containerID="cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9"
Oct 30 19:48:49.638209 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.638182    6777 generic.go:356] "Generic (PLEG): container finished" podID="f54a1f71-c71b-402c-b7a5-14859066df28" containerID="884decb53796c3f4c0a50218e289e2ce5988860ea2a1d379e35c256064d9009d" exitCode=0
Oct 30 19:48:49.638349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.638270    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f" event={"ID":"f54a1f71-c71b-402c-b7a5-14859066df28","Type":"ContainerDied","Data":"884decb53796c3f4c0a50218e289e2ce5988860ea2a1d379e35c256064d9009d"}
Oct 30 19:48:49.639874 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.639847    6777 generic.go:356] "Generic (PLEG): container finished" podID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerID="0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c" exitCode=0
Oct 30 19:48:49.640013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.639879    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" event={"ID":"b7ded767-98c6-459e-9209-3c58dc4751c8","Type":"ContainerDied","Data":"0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c"}
Oct 30 19:48:49.640013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.639894    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"
Oct 30 19:48:49.640013 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.639924    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr" event={"ID":"b7ded767-98c6-459e-9209-3c58dc4751c8","Type":"ContainerDied","Data":"2eec1a5d2b562f02ce02b78e113a617b341d2536dfbbf9f9d3cd2ec431912b88"}
Oct 30 19:48:49.648104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.648085    6777 scope.go:122] "RemoveContainer" containerID="cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9"
Oct 30 19:48:49.648402 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:49.648379    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9\": container with ID starting with cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9 not found: ID does not exist" containerID="cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9"
Oct 30 19:48:49.648473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.648411    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9"} err="failed to get container status \"cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9\": rpc error: code = NotFound desc = could not find container \"cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9\": container with ID starting with cdd3d9b75fc1269bdd2238206dd9187fcf913bf7e0aea716f6629ef702df3cc9 not found: ID does not exist"
Oct 30 19:48:49.648473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.648434    6777 scope.go:122] "RemoveContainer" containerID="0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c"
Oct 30 19:48:49.660358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.660331    6777 scope.go:122] "RemoveContainer" containerID="0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c"
Oct 30 19:48:49.660680 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:49.660658    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c\": container with ID starting with 0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c not found: ID does not exist" containerID="0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c"
Oct 30 19:48:49.660725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.660689    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c"} err="failed to get container status \"0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c\": rpc error: code = NotFound desc = could not find container \"0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c\": container with ID starting with 0f3624d9b895fe4f83376e4e95fe61522eba942bdb6ed91227a3f560f34ddd0c not found: ID does not exist"
Oct 30 19:48:49.755931 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.755899    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"]
Oct 30 19:48:49.758352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.758327    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-l5mvb"]
Oct 30 19:48:49.811462 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.811424    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"]
Oct 30 19:48:49.831115 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.831078    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-hv7rr"]
Oct 30 19:48:49.891192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:49.889172    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:48:50.113694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:50.113654    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" path="/var/lib/kubelet/pods/96ca458c-b3a8-4c14-93b5-4da753fbc136/volumes"
Oct 30 19:48:50.114190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:50.114175    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" path="/var/lib/kubelet/pods/b7ded767-98c6-459e-9209-3c58dc4751c8/volumes"
Oct 30 19:48:50.803191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:50.803161    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:50.945174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:50.945081    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54a1f71-c71b-402c-b7a5-14859066df28-kube-api-access-qjjfw\" (UniqueName: \"kubernetes.io/projected/f54a1f71-c71b-402c-b7a5-14859066df28-kube-api-access-qjjfw\") pod \"f54a1f71-c71b-402c-b7a5-14859066df28\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") "
Oct 30 19:48:50.945174 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:50.945146    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-xks-ca\") pod \"f54a1f71-c71b-402c-b7a5-14859066df28\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") "
Oct 30 19:48:50.945409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:50.945189    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-openshift-ca\") pod \"f54a1f71-c71b-402c-b7a5-14859066df28\" (UID: \"f54a1f71-c71b-402c-b7a5-14859066df28\") "
Oct 30 19:48:50.945697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:50.945668    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-openshift-ca" pod "f54a1f71-c71b-402c-b7a5-14859066df28" (UID: "f54a1f71-c71b-402c-b7a5-14859066df28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:50.947427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:50.947396    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54a1f71-c71b-402c-b7a5-14859066df28-kube-api-access-qjjfw" pod "f54a1f71-c71b-402c-b7a5-14859066df28" (UID: "f54a1f71-c71b-402c-b7a5-14859066df28"). InnerVolumeSpecName "kube-api-access-qjjfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:50.947526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:50.947477    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-xks-ca" pod "f54a1f71-c71b-402c-b7a5-14859066df28" (UID: "f54a1f71-c71b-402c-b7a5-14859066df28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:51.046995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:51.046954    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjfw\" (UniqueName: \"kubernetes.io/projected/f54a1f71-c71b-402c-b7a5-14859066df28-kube-api-access-qjjfw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.046995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:51.046993    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.046995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:51.047002    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54a1f71-c71b-402c-b7a5-14859066df28-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.656922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:51.656881    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f" event={"ID":"f54a1f71-c71b-402c-b7a5-14859066df28","Type":"ContainerDied","Data":"cf34f83ede7e54b44b68a45c6c02c4604315093b43e6a48345890d12fbf11a10"}
Oct 30 19:48:51.656922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:51.656922    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"
Oct 30 19:48:51.657178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:51.656945    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf34f83ede7e54b44b68a45c6c02c4604315093b43e6a48345890d12fbf11a10"
Oct 30 19:48:51.838504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:51.838462    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp" podUID="f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:51.838905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:51.838462    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx" podUID="b9dcef83-3bad-4f42-81a8-3051ec762234" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:53.813052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:53.813015    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"]
Oct 30 19:48:53.813632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:53.813244    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy" containerID="cri-o://5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a" gracePeriod=30
Oct 30 19:48:57.172627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:57.172583    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp" podUID="e67fd308-ff22-476f-82a1-e22d41c8381e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:57.173116 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:57.173023    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr" podUID="095c2e53-0ad0-40f6-af86-0bad69490983" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:57.242480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:57.242440    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"]
Oct 30 19:48:59.183163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.183125    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:48:59.218005 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.217974    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:48:59.251735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.251705    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-certs\") pod \"c07861c1-e442-495a-9e32-231b53ae1c5d\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") "
Oct 30 19:48:59.251938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.251781    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-podinfo\") pod \"c07861c1-e442-495a-9e32-231b53ae1c5d\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") "
Oct 30 19:48:59.251938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.251866    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-kube-api-access-v7vq5\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-kube-api-access-v7vq5\") pod \"c07861c1-e442-495a-9e32-231b53ae1c5d\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") "
Oct 30 19:48:59.251938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.251907    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-token\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-token\") pod \"c07861c1-e442-495a-9e32-231b53ae1c5d\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") "
Oct 30 19:48:59.251938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.251933    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-credential-socket\") pod \"c07861c1-e442-495a-9e32-231b53ae1c5d\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") "
Oct 30 19:48:59.252249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.251967    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-envoy\") pod \"c07861c1-e442-495a-9e32-231b53ae1c5d\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") "
Oct 30 19:48:59.252249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.251991    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-plugins-config-volume\") pod \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\" (UID: \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\") "
Oct 30 19:48:59.252249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252006    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-certs" pod "c07861c1-e442-495a-9e32-231b53ae1c5d" (UID: "c07861c1-e442-495a-9e32-231b53ae1c5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.252249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252024    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07861c1-e442-495a-9e32-231b53ae1c5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07861c1-e442-495a-9e32-231b53ae1c5d-istiod-ca-cert\") pod \"c07861c1-e442-495a-9e32-231b53ae1c5d\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") "
Oct 30 19:48:59.252249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252041    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-socket\") pod \"c07861c1-e442-495a-9e32-231b53ae1c5d\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") "
Oct 30 19:48:59.252249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252132    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-data\") pod \"c07861c1-e442-495a-9e32-231b53ae1c5d\" (UID: \"c07861c1-e442-495a-9e32-231b53ae1c5d\") "
Oct 30 19:48:59.252249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252198    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-kube-api-access-nq5dv\" (UniqueName: \"kubernetes.io/projected/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-kube-api-access-nq5dv\") pod \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\" (UID: \"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2\") "
Oct 30 19:48:59.252654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252634    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-socket" pod "c07861c1-e442-495a-9e32-231b53ae1c5d" (UID: "c07861c1-e442-495a-9e32-231b53ae1c5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.252731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252677    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.252892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252867    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-data" pod "c07861c1-e442-495a-9e32-231b53ae1c5d" (UID: "c07861c1-e442-495a-9e32-231b53ae1c5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.252892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252881    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-plugins-config-volume" pod "f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" (UID: "f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:59.253026 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.252878    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-credential-socket" pod "c07861c1-e442-495a-9e32-231b53ae1c5d" (UID: "c07861c1-e442-495a-9e32-231b53ae1c5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.253138 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.253077    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07861c1-e442-495a-9e32-231b53ae1c5d-istiod-ca-cert" pod "c07861c1-e442-495a-9e32-231b53ae1c5d" (UID: "c07861c1-e442-495a-9e32-231b53ae1c5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:59.255649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.255621    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:48:59.257230 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.257207    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-kube-api-access-v7vq5" pod "c07861c1-e442-495a-9e32-231b53ae1c5d" (UID: "c07861c1-e442-495a-9e32-231b53ae1c5d"). InnerVolumeSpecName "kube-api-access-v7vq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:59.263615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.263584    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-podinfo" pod "c07861c1-e442-495a-9e32-231b53ae1c5d" (UID: "c07861c1-e442-495a-9e32-231b53ae1c5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:59.263726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.263629    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-token" pod "c07861c1-e442-495a-9e32-231b53ae1c5d" (UID: "c07861c1-e442-495a-9e32-231b53ae1c5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:59.263726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.263666    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-kube-api-access-nq5dv" pod "f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" (UID: "f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2"). InnerVolumeSpecName "kube-api-access-nq5dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:59.263810 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.263767    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-envoy" pod "c07861c1-e442-495a-9e32-231b53ae1c5d" (UID: "c07861c1-e442-495a-9e32-231b53ae1c5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.353821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.353778    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9dcef83-3bad-4f42-81a8-3051ec762234-kube-api-access-fnk4f\" (UniqueName: \"kubernetes.io/projected/b9dcef83-3bad-4f42-81a8-3051ec762234-kube-api-access-fnk4f\") pod \"b9dcef83-3bad-4f42-81a8-3051ec762234\" (UID: \"b9dcef83-3bad-4f42-81a8-3051ec762234\") "
Oct 30 19:48:59.353821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.353823    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9dcef83-3bad-4f42-81a8-3051ec762234-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dcef83-3bad-4f42-81a8-3051ec762234-plugins-config-volume\") pod \"b9dcef83-3bad-4f42-81a8-3051ec762234\" (UID: \"b9dcef83-3bad-4f42-81a8-3051ec762234\") "
Oct 30 19:48:59.354343 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354315    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9dcef83-3bad-4f42-81a8-3051ec762234-plugins-config-volume" pod "b9dcef83-3bad-4f42-81a8-3051ec762234" (UID: "b9dcef83-3bad-4f42-81a8-3051ec762234"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:59.354468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354444    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7vq5\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-kube-api-access-v7vq5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354461    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354470    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dcef83-3bad-4f42-81a8-3051ec762234-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354480    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354492    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354500    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354508    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07861c1-e442-495a-9e32-231b53ae1c5d-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354518    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354526    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354534    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq5dv\" (UniqueName: \"kubernetes.io/projected/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2-kube-api-access-nq5dv\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.354547 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.354543    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07861c1-e442-495a-9e32-231b53ae1c5d-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.356125 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.356093    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9dcef83-3bad-4f42-81a8-3051ec762234-kube-api-access-fnk4f" pod "b9dcef83-3bad-4f42-81a8-3051ec762234" (UID: "b9dcef83-3bad-4f42-81a8-3051ec762234"). InnerVolumeSpecName "kube-api-access-fnk4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:59.455576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.455477    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnk4f\" (UniqueName: \"kubernetes.io/projected/b9dcef83-3bad-4f42-81a8-3051ec762234-kube-api-access-fnk4f\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.599022    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"]
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600558    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f54a1f71-c71b-402c-b7a5-14859066df28" containerName="revoke-keys"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600583    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54a1f71-c71b-402c-b7a5-14859066df28" containerName="revoke-keys"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600598    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b9dcef83-3bad-4f42-81a8-3051ec762234" containerName="payload-pre-processing"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600622    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9dcef83-3bad-4f42-81a8-3051ec762234" containerName="payload-pre-processing"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600733    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600742    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600787    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" containerName="payload-processing"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600796    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" containerName="payload-processing"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600813    6777 cpu_manager.go:414] "containerMap: removing container" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600820    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600856    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.600864    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.601121    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ded767-98c6-459e-9209-3c58dc4751c8" containerName="istio-proxy"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.601148    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" containerName="payload-processing"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.601162    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ca458c-b3a8-4c14-93b5-4da753fbc136" containerName="istio-proxy"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.601180    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerName="istio-proxy"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.601200    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54a1f71-c71b-402c-b7a5-14859066df28" containerName="revoke-keys"
Oct 30 19:48:59.601325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.601208    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9dcef83-3bad-4f42-81a8-3051ec762234" containerName="payload-pre-processing"
Oct 30 19:48:59.608878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.608846    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.616459 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.616432    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"]
Oct 30 19:48:59.658617 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.658578    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb88e92-4nq4p\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.658844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.658734    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb88e92-4nq4p\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.658844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.658805    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zc6\" (UniqueName: \"kubernetes.io/projected/7ffe77ce-28ce-4512-a13b-f46175c14dc4-kube-api-access-r5zc6\") pod \"maas-api-revoke-sub-0bb88e92-4nq4p\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.702284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.702248    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:59.702482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.702339    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:59.702482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.702401    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:59.702482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.702439    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:59.714935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.714842    6777 generic.go:356] "Generic (PLEG): container finished" podID="c07861c1-e442-495a-9e32-231b53ae1c5d" containerID="5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a" exitCode=0
Oct 30 19:48:59.714935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.714922    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"
Oct 30 19:48:59.714935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.714923    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" event={"ID":"c07861c1-e442-495a-9e32-231b53ae1c5d","Type":"ContainerDied","Data":"5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a"}
Oct 30 19:48:59.715281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.714988    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6" event={"ID":"c07861c1-e442-495a-9e32-231b53ae1c5d","Type":"ContainerDied","Data":"6a45b23b8e30679e1c5bbc95a4560bac873e489217be84d30409984326bd320d"}
Oct 30 19:48:59.715281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.715012    6777 scope.go:122] "RemoveContainer" containerID="5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a"
Oct 30 19:48:59.716643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.716621    6777 generic.go:356] "Generic (PLEG): container finished" podID="f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" containerID="16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f" exitCode=0
Oct 30 19:48:59.716764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.716654    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp" event={"ID":"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2","Type":"ContainerDied","Data":"16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f"}
Oct 30 19:48:59.716764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.716694    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"
Oct 30 19:48:59.716764 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.716724    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp" event={"ID":"f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2","Type":"ContainerDied","Data":"dd9157752a133578e70687d21283741378a738357a596b00b8fef6a5123feb08"}
Oct 30 19:48:59.718277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.718253    6777 generic.go:356] "Generic (PLEG): container finished" podID="b9dcef83-3bad-4f42-81a8-3051ec762234" containerID="c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81" exitCode=0
Oct 30 19:48:59.718414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.718311    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"
Oct 30 19:48:59.718414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.718344    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx" event={"ID":"b9dcef83-3bad-4f42-81a8-3051ec762234","Type":"ContainerDied","Data":"c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81"}
Oct 30 19:48:59.718414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.718387    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx" event={"ID":"b9dcef83-3bad-4f42-81a8-3051ec762234","Type":"ContainerDied","Data":"5fc218c33a1e11ec14c77ddb1e6f506cd0cc2d7d7a987f89d9634670d3811221"}
Oct 30 19:48:59.730576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.730552    6777 scope.go:122] "RemoveContainer" containerID="5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a"
Oct 30 19:48:59.731115 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:59.731051    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a\": container with ID starting with 5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a not found: ID does not exist" containerID="5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a"
Oct 30 19:48:59.731232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.731125    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a"} err="failed to get container status \"5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a\": rpc error: code = NotFound desc = could not find container \"5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a\": container with ID starting with 5300da90d76b709de3bcceb111d6a3a2bbeac203c3b6151ef2f830187d8a8b0a not found: ID does not exist"
Oct 30 19:48:59.731232 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.731146    6777 scope.go:122] "RemoveContainer" containerID="16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f"
Oct 30 19:48:59.741105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.741075    6777 scope.go:122] "RemoveContainer" containerID="16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f"
Oct 30 19:48:59.741427 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:59.741406    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f\": container with ID starting with 16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f not found: ID does not exist" containerID="16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f"
Oct 30 19:48:59.741508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.741439    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f"} err="failed to get container status \"16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f\": rpc error: code = NotFound desc = could not find container \"16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f\": container with ID starting with 16c344bece258c4912313332d1aa5c2791193f1a0b3dc4522c61e5179739f56f not found: ID does not exist"
Oct 30 19:48:59.741508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.741461    6777 scope.go:122] "RemoveContainer" containerID="c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81"
Oct 30 19:48:59.749936 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.749905    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"]
Oct 30 19:48:59.752844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.752821    6777 scope.go:122] "RemoveContainer" containerID="c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81"
Oct 30 19:48:59.753261 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:48:59.753241    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81\": container with ID starting with c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81 not found: ID does not exist" containerID="c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81"
Oct 30 19:48:59.753325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.753270    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81"} err="failed to get container status \"c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81\": rpc error: code = NotFound desc = could not find container \"c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81\": container with ID starting with c965fc178b4558610a59336c4af5e0b663da6a928c2b1c8a5ec21dd9897efe81 not found: ID does not exist"
Oct 30 19:48:59.754924 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.754900    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8f3ff4-849b4fdb59-q8nxx"]
Oct 30 19:48:59.760584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.760555    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb88e92-4nq4p\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.760687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.760652    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb88e92-4nq4p\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.760687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.760669    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zc6\" (UniqueName: \"kubernetes.io/projected/7ffe77ce-28ce-4512-a13b-f46175c14dc4-kube-api-access-r5zc6\") pod \"maas-api-revoke-sub-0bb88e92-4nq4p\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.761246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.761227    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb88e92-4nq4p\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.762882 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.762865    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb88e92-4nq4p\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.767387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.767360    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"]
Oct 30 19:48:59.773782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.773754    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8f3ff4-89c77ff84-nszrp"]
Oct 30 19:48:59.775441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.775412    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zc6\" (UniqueName: \"kubernetes.io/projected/7ffe77ce-28ce-4512-a13b-f46175c14dc4-kube-api-access-r5zc6\") pod \"maas-api-revoke-sub-0bb88e92-4nq4p\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:48:59.788334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.788303    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"]
Oct 30 19:48:59.793798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.793765    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-whjp6"]
Oct 30 19:48:59.927423 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:48:59.927389    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:49:00.112098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:00.112018    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9dcef83-3bad-4f42-81a8-3051ec762234" path="/var/lib/kubelet/pods/b9dcef83-3bad-4f42-81a8-3051ec762234/volumes"
Oct 30 19:49:00.112660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:00.112635    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07861c1-e442-495a-9e32-231b53ae1c5d" path="/var/lib/kubelet/pods/c07861c1-e442-495a-9e32-231b53ae1c5d/volumes"
Oct 30 19:49:00.113186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:00.113164    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2" path="/var/lib/kubelet/pods/f93c51db-f93e-4cf5-aaa9-1a74a9b55fe2/volumes"
Oct 30 19:49:00.210318 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:00.210293    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"]
Oct 30 19:49:00.212472 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:49:00.212443    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffe77ce_28ce_4512_a13b_f46175c14dc4.slice/crio-90a80a9adc0264e491a8748d456d1ed902ca00fbd02b80833d7dcaea37ba8e52 WatchSource:0}: Error finding container 90a80a9adc0264e491a8748d456d1ed902ca00fbd02b80833d7dcaea37ba8e52: Status 404 returned error can't find the container with id 90a80a9adc0264e491a8748d456d1ed902ca00fbd02b80833d7dcaea37ba8e52
Oct 30 19:49:00.726244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:00.726203    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p" event={"ID":"7ffe77ce-28ce-4512-a13b-f46175c14dc4","Type":"ContainerStarted","Data":"8e5652556fbc732c5be138ab839820b58851415b106178a326e52adacba25987"}
Oct 30 19:49:00.726433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:00.726277    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p" event={"ID":"7ffe77ce-28ce-4512-a13b-f46175c14dc4","Type":"ContainerStarted","Data":"90a80a9adc0264e491a8748d456d1ed902ca00fbd02b80833d7dcaea37ba8e52"}
Oct 30 19:49:00.746541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:00.746488    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p" podStartSLOduration=1.746471429 podStartE2EDuration="1.746471429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:00.744709003 +0000 UTC m=+2522.741624388" watchObservedRunningTime="2026-10-30 19:49:00.746471429 +0000 UTC m=+2522.743386813"
Oct 30 19:49:01.738683 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:01.738641    6777 generic.go:356] "Generic (PLEG): container finished" podID="7ffe77ce-28ce-4512-a13b-f46175c14dc4" containerID="8e5652556fbc732c5be138ab839820b58851415b106178a326e52adacba25987" exitCode=0
Oct 30 19:49:01.739141 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:01.738788    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p" event={"ID":"7ffe77ce-28ce-4512-a13b-f46175c14dc4","Type":"ContainerDied","Data":"8e5652556fbc732c5be138ab839820b58851415b106178a326e52adacba25987"}
Oct 30 19:49:02.242886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:02.242843    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6" podUID="52606f4d-8ec1-42d9-b49a-67218a118f6f" containerName="payload-processing" containerID="cri-o://e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b" gracePeriod=25
Oct 30 19:49:02.892126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:02.892100    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:49:03.001476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.001368    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ffe77ce-28ce-4512-a13b-f46175c14dc4-kube-api-access-r5zc6\" (UniqueName: \"kubernetes.io/projected/7ffe77ce-28ce-4512-a13b-f46175c14dc4-kube-api-access-r5zc6\") pod \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") "
Oct 30 19:49:03.001476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.001440    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-openshift-ca\") pod \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") "
Oct 30 19:49:03.001729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.001559    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-xks-ca\") pod \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\" (UID: \"7ffe77ce-28ce-4512-a13b-f46175c14dc4\") "
Oct 30 19:49:03.001867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.001843    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-openshift-ca" pod "7ffe77ce-28ce-4512-a13b-f46175c14dc4" (UID: "7ffe77ce-28ce-4512-a13b-f46175c14dc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:03.002184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.002154    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.003664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.003645    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-xks-ca" pod "7ffe77ce-28ce-4512-a13b-f46175c14dc4" (UID: "7ffe77ce-28ce-4512-a13b-f46175c14dc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:03.003797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.003783    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffe77ce-28ce-4512-a13b-f46175c14dc4-kube-api-access-r5zc6" pod "7ffe77ce-28ce-4512-a13b-f46175c14dc4" (UID: "7ffe77ce-28ce-4512-a13b-f46175c14dc4"). InnerVolumeSpecName "kube-api-access-r5zc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:03.103564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.103531    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffe77ce-28ce-4512-a13b-f46175c14dc4-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.103564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.103560    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5zc6\" (UniqueName: \"kubernetes.io/projected/7ffe77ce-28ce-4512-a13b-f46175c14dc4-kube-api-access-r5zc6\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.758074 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.758034    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"
Oct 30 19:49:03.758243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.758104    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p" event={"ID":"7ffe77ce-28ce-4512-a13b-f46175c14dc4","Type":"ContainerDied","Data":"90a80a9adc0264e491a8748d456d1ed902ca00fbd02b80833d7dcaea37ba8e52"}
Oct 30 19:49:03.758243 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:03.758152    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a80a9adc0264e491a8748d456d1ed902ca00fbd02b80833d7dcaea37ba8e52"
Oct 30 19:49:04.524662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:04.524615    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"]
Oct 30 19:49:06.196962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:06.196925    6777 scope.go:122] "RemoveContainer" containerID="5ecd70bd2b17f039dfdb201471302cf2818a86d51b32d3b7140f05e7f14fd2a6"
Oct 30 19:49:07.172258 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.172222    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr" podUID="095c2e53-0ad0-40f6-af86-0bad69490983" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:49:07.172433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.172233    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp" podUID="e67fd308-ff22-476f-82a1-e22d41c8381e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:49:07.441687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.441611    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:49:07.492100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.492046    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:49:07.530761 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.530720    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6" podUID="52606f4d-8ec1-42d9-b49a-67218a118f6f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:49:07.555594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.555550    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095c2e53-0ad0-40f6-af86-0bad69490983-kube-api-access-cfn78\" (UniqueName: \"kubernetes.io/projected/095c2e53-0ad0-40f6-af86-0bad69490983-kube-api-access-cfn78\") pod \"095c2e53-0ad0-40f6-af86-0bad69490983\" (UID: \"095c2e53-0ad0-40f6-af86-0bad69490983\") "
Oct 30 19:49:07.555782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.555627    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67fd308-ff22-476f-82a1-e22d41c8381e-kube-api-access-rz45s\" (UniqueName: \"kubernetes.io/projected/e67fd308-ff22-476f-82a1-e22d41c8381e-kube-api-access-rz45s\") pod \"e67fd308-ff22-476f-82a1-e22d41c8381e\" (UID: \"e67fd308-ff22-476f-82a1-e22d41c8381e\") "
Oct 30 19:49:07.555782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.555718    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095c2e53-0ad0-40f6-af86-0bad69490983-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c2e53-0ad0-40f6-af86-0bad69490983-plugins-config-volume\") pod \"095c2e53-0ad0-40f6-af86-0bad69490983\" (UID: \"095c2e53-0ad0-40f6-af86-0bad69490983\") "
Oct 30 19:49:07.555932 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.555868    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e67fd308-ff22-476f-82a1-e22d41c8381e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e67fd308-ff22-476f-82a1-e22d41c8381e-plugins-config-volume\") pod \"e67fd308-ff22-476f-82a1-e22d41c8381e\" (UID: \"e67fd308-ff22-476f-82a1-e22d41c8381e\") "
Oct 30 19:49:07.556159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.556124    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095c2e53-0ad0-40f6-af86-0bad69490983-plugins-config-volume" pod "095c2e53-0ad0-40f6-af86-0bad69490983" (UID: "095c2e53-0ad0-40f6-af86-0bad69490983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:07.556295 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.556193    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67fd308-ff22-476f-82a1-e22d41c8381e-plugins-config-volume" pod "e67fd308-ff22-476f-82a1-e22d41c8381e" (UID: "e67fd308-ff22-476f-82a1-e22d41c8381e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:07.556525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.556506    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c2e53-0ad0-40f6-af86-0bad69490983-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.556604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.556536    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e67fd308-ff22-476f-82a1-e22d41c8381e-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.557967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.557939    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67fd308-ff22-476f-82a1-e22d41c8381e-kube-api-access-rz45s" pod "e67fd308-ff22-476f-82a1-e22d41c8381e" (UID: "e67fd308-ff22-476f-82a1-e22d41c8381e"). InnerVolumeSpecName "kube-api-access-rz45s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:07.558170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.557978    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095c2e53-0ad0-40f6-af86-0bad69490983-kube-api-access-cfn78" pod "095c2e53-0ad0-40f6-af86-0bad69490983" (UID: "095c2e53-0ad0-40f6-af86-0bad69490983"). InnerVolumeSpecName "kube-api-access-cfn78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:07.658194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.658153    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfn78\" (UniqueName: \"kubernetes.io/projected/095c2e53-0ad0-40f6-af86-0bad69490983-kube-api-access-cfn78\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.658194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.658189    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz45s\" (UniqueName: \"kubernetes.io/projected/e67fd308-ff22-476f-82a1-e22d41c8381e-kube-api-access-rz45s\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.851829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.851794    6777 generic.go:356] "Generic (PLEG): container finished" podID="095c2e53-0ad0-40f6-af86-0bad69490983" containerID="976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58" exitCode=0
Oct 30 19:49:07.852006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.851845    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"
Oct 30 19:49:07.852006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.851879    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr" event={"ID":"095c2e53-0ad0-40f6-af86-0bad69490983","Type":"ContainerDied","Data":"976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58"}
Oct 30 19:49:07.852006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.851932    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr" event={"ID":"095c2e53-0ad0-40f6-af86-0bad69490983","Type":"ContainerDied","Data":"3eb77e35976bed2489a9f017075d6fbe4e32984be2557d989ac183c7f6b4320e"}
Oct 30 19:49:07.852006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.851959    6777 scope.go:122] "RemoveContainer" containerID="976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58"
Oct 30 19:49:07.854317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.854291    6777 generic.go:356] "Generic (PLEG): container finished" podID="e67fd308-ff22-476f-82a1-e22d41c8381e" containerID="32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8" exitCode=0
Oct 30 19:49:07.854447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.854344    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"
Oct 30 19:49:07.854447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.854376    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp" event={"ID":"e67fd308-ff22-476f-82a1-e22d41c8381e","Type":"ContainerDied","Data":"32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8"}
Oct 30 19:49:07.854447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.854418    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp" event={"ID":"e67fd308-ff22-476f-82a1-e22d41c8381e","Type":"ContainerDied","Data":"b7f21168abac1771f55d7a9b85d68b8c7372b2fd09329bda7a2c1f105288620b"}
Oct 30 19:49:07.864575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.864550    6777 scope.go:122] "RemoveContainer" containerID="976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58"
Oct 30 19:49:07.864903 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:07.864880    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58\": container with ID starting with 976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58 not found: ID does not exist" containerID="976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58"
Oct 30 19:49:07.865003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.864913    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58"} err="failed to get container status \"976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58\": rpc error: code = NotFound desc = could not find container \"976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58\": container with ID starting with 976395df58fdbdca57b4b3e1bd870bbf63694ac88f4a90c521526297af5efb58 not found: ID does not exist"
Oct 30 19:49:07.865003 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.864931    6777 scope.go:122] "RemoveContainer" containerID="32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8"
Oct 30 19:49:07.875508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.875478    6777 scope.go:122] "RemoveContainer" containerID="32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8"
Oct 30 19:49:07.875844 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:07.875823    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8\": container with ID starting with 32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8 not found: ID does not exist" containerID="32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8"
Oct 30 19:49:07.876008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.875851    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8"} err="failed to get container status \"32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8\": rpc error: code = NotFound desc = could not find container \"32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8\": container with ID starting with 32813219e067eb2d79c6eaf1b48a21d71efedb2b8e035565668632c9f93319f8 not found: ID does not exist"
Oct 30 19:49:07.880078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.880037    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"]
Oct 30 19:49:07.887387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.887355    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ddb6ea-75564d979-6j2pr"]
Oct 30 19:49:07.901041 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.901000    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"]
Oct 30 19:49:07.903095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:07.903074    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ddb6ea-849d9cf6b-87fsp"]
Oct 30 19:49:08.113851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:08.113771    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095c2e53-0ad0-40f6-af86-0bad69490983" path="/var/lib/kubelet/pods/095c2e53-0ad0-40f6-af86-0bad69490983/volumes"
Oct 30 19:49:08.114251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:08.114237    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67fd308-ff22-476f-82a1-e22d41c8381e" path="/var/lib/kubelet/pods/e67fd308-ff22-476f-82a1-e22d41c8381e/volumes"
Oct 30 19:49:08.670391 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:08.670346    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:49:09.525725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:09.525654    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn" podUID="ba70864a-66bd-44b6-9923-0ac9f00260fd" containerName="payload-processing" containerID="cri-o://7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f" gracePeriod=25
Oct 30 19:49:13.582273 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:13.582238    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"]
Oct 30 19:49:13.586371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:13.586341    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39b867-xnh8f"]
Oct 30 19:49:13.639628 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:13.639586    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:49:14.116756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:14.116722    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54a1f71-c71b-402c-b7a5-14859066df28" path="/var/lib/kubelet/pods/f54a1f71-c71b-402c-b7a5-14859066df28/volumes"
Oct 30 19:49:14.441208 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:14.441111    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"]
Oct 30 19:49:14.442244 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:14.442185    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy" containerID="cri-o://5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9" gracePeriod=30
Oct 30 19:49:14.447228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:14.447187    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"]
Oct 30 19:49:14.447647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:14.447593    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy" containerID="cri-o://028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8" gracePeriod=30
Oct 30 19:49:14.452490 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:14.452459    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"]
Oct 30 19:49:14.452801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:14.452740    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy" containerID="cri-o://10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe" gracePeriod=30
Oct 30 19:49:14.460390 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:14.460359    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"]
Oct 30 19:49:14.460619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:14.460593    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" containerName="istio-proxy" containerID="cri-o://b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec" gracePeriod=30
Oct 30 19:49:16.539081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:16.539009    6777 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:16.539493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:16.539109    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:16.966503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:16.966411    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:49:17.528460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:17.528420    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6" podUID="52606f4d-8ec1-42d9-b49a-67218a118f6f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:49:18.031186 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:18.031145    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"]
Oct 30 19:49:18.038851 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:18.038817    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe14008-q5sz8"]
Oct 30 19:49:18.112213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:18.112177    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30def2e4-b284-4a35-97f7-4f92fdc1920d" path="/var/lib/kubelet/pods/30def2e4-b284-4a35-97f7-4f92fdc1920d/volumes"
Oct 30 19:49:18.652909 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:18.652870    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn" podUID="ba70864a-66bd-44b6-9923-0ac9f00260fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:49:19.748859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.748827    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:49:19.792096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.791552    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:49:19.830448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.830422    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:49:19.842509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.842480    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:49:19.909253 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909220    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-credential-socket\") pod \"4405b1c4-1a26-4592-a367-e763601cba0c\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") "
Oct 30 19:49:19.909464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909265    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-certs\") pod \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") "
Oct 30 19:49:19.909464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909299    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4405b1c4-1a26-4592-a367-e763601cba0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4405b1c4-1a26-4592-a367-e763601cba0c-istiod-ca-cert\") pod \"4405b1c4-1a26-4592-a367-e763601cba0c\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") "
Oct 30 19:49:19.909464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909324    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-certs\") pod \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") "
Oct 30 19:49:19.909464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909351    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-envoy\") pod \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") "
Oct 30 19:49:19.909464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909390    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-token\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-token\") pod \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") "
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909463    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istiod-ca-cert\") pod \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") "
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909503    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-podinfo\") pod \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") "
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909532    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4405b1c4-1a26-4592-a367-e763601cba0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4405b1c4-1a26-4592-a367-e763601cba0c-istio-podinfo\") pod \"4405b1c4-1a26-4592-a367-e763601cba0c\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") "
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909578    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-socket\") pod \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") "
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909582    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-credential-socket" pod "4405b1c4-1a26-4592-a367-e763601cba0c" (UID: "4405b1c4-1a26-4592-a367-e763601cba0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909593    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-certs" pod "1a01e0ca-b6ee-44c6-be57-f94a25d39f21" (UID: "1a01e0ca-b6ee-44c6-be57-f94a25d39f21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909605    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-certs\") pod \"4405b1c4-1a26-4592-a367-e763601cba0c\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") "
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909632    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-token\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-token\") pod \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") "
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909673    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-data\") pod \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") "
Oct 30 19:49:19.909758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909701    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-kube-api-access-rhw4v\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-kube-api-access-rhw4v\") pod \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909774    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-envoy\") pod \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909827    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-data\") pod \"4405b1c4-1a26-4592-a367-e763601cba0c\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909874    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-kube-api-access-w6qrl\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-kube-api-access-w6qrl\") pod \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909903    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-socket\") pod \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909927    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-istio-token\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-istio-token\") pod \"4405b1c4-1a26-4592-a367-e763601cba0c\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909953    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-credential-socket\") pod \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909980    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-kube-api-access-62dtt\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-kube-api-access-62dtt\") pod \"4405b1c4-1a26-4592-a367-e763601cba0c\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910042    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-credential-socket\") pod \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910096    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-podinfo\") pod \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910137    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istiod-ca-cert\") pod \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\" (UID: \"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910168    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-data\") pod \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\" (UID: \"1a01e0ca-b6ee-44c6-be57-f94a25d39f21\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910211    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-envoy\") pod \"4405b1c4-1a26-4592-a367-e763601cba0c\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") "
Oct 30 19:49:19.910369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910239    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-socket\") pod \"4405b1c4-1a26-4592-a367-e763601cba0c\" (UID: \"4405b1c4-1a26-4592-a367-e763601cba0c\") "
Oct 30 19:49:19.911201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910855    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.911201 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910877    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.911756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909670    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-certs" pod "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" (UID: "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.911894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909824    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4405b1c4-1a26-4592-a367-e763601cba0c-istiod-ca-cert" pod "4405b1c4-1a26-4592-a367-e763601cba0c" (UID: "4405b1c4-1a26-4592-a367-e763601cba0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:19.911894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909874    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-socket" pod "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" (UID: "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.911894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.909905    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istiod-ca-cert" pod "1a01e0ca-b6ee-44c6-be57-f94a25d39f21" (UID: "1a01e0ca-b6ee-44c6-be57-f94a25d39f21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:19.911894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910007    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-data" pod "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" (UID: "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.911894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910048    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-certs" pod "4405b1c4-1a26-4592-a367-e763601cba0c" (UID: "4405b1c4-1a26-4592-a367-e763601cba0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.911894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.910309    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-data" pod "4405b1c4-1a26-4592-a367-e763601cba0c" (UID: "4405b1c4-1a26-4592-a367-e763601cba0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.911894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.911793    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4405b1c4-1a26-4592-a367-e763601cba0c-istio-podinfo" pod "4405b1c4-1a26-4592-a367-e763601cba0c" (UID: "4405b1c4-1a26-4592-a367-e763601cba0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:19.911894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.911119    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-socket" pod "4405b1c4-1a26-4592-a367-e763601cba0c" (UID: "4405b1c4-1a26-4592-a367-e763601cba0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.912393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.911958    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-credential-socket" pod "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" (UID: "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.912393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.911986    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-socket" pod "1a01e0ca-b6ee-44c6-be57-f94a25d39f21" (UID: "1a01e0ca-b6ee-44c6-be57-f94a25d39f21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.912393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.912036    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-credential-socket" pod "1a01e0ca-b6ee-44c6-be57-f94a25d39f21" (UID: "1a01e0ca-b6ee-44c6-be57-f94a25d39f21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.912393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.912324    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-data" pod "1a01e0ca-b6ee-44c6-be57-f94a25d39f21" (UID: "1a01e0ca-b6ee-44c6-be57-f94a25d39f21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.912393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.912351    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istiod-ca-cert" pod "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" (UID: "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:19.912699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.912673    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-token" pod "1a01e0ca-b6ee-44c6-be57-f94a25d39f21" (UID: "1a01e0ca-b6ee-44c6-be57-f94a25d39f21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.912821 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.912791    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-podinfo" pod "1a01e0ca-b6ee-44c6-be57-f94a25d39f21" (UID: "1a01e0ca-b6ee-44c6-be57-f94a25d39f21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:19.912971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.912931    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-envoy" pod "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" (UID: "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.913137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.913114    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-kube-api-access-rhw4v" pod "1a01e0ca-b6ee-44c6-be57-f94a25d39f21" (UID: "1a01e0ca-b6ee-44c6-be57-f94a25d39f21"). InnerVolumeSpecName "kube-api-access-rhw4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.914126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.914053    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-kube-api-access-w6qrl" pod "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" (UID: "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce"). InnerVolumeSpecName "kube-api-access-w6qrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.914867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.914618    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-token" pod "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" (UID: "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.914867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.914709    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-istio-token" pod "4405b1c4-1a26-4592-a367-e763601cba0c" (UID: "4405b1c4-1a26-4592-a367-e763601cba0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.915113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.915090    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-podinfo" pod "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" (UID: "4eb6c1cf-e34a-49af-b7e6-9080d5c699ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:19.915234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.915211    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-envoy" pod "1a01e0ca-b6ee-44c6-be57-f94a25d39f21" (UID: "1a01e0ca-b6ee-44c6-be57-f94a25d39f21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.915719 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.915688    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-envoy" pod "4405b1c4-1a26-4592-a367-e763601cba0c" (UID: "4405b1c4-1a26-4592-a367-e763601cba0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.916485 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.916468    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-kube-api-access-62dtt" pod "4405b1c4-1a26-4592-a367-e763601cba0c" (UID: "4405b1c4-1a26-4592-a367-e763601cba0c"). InnerVolumeSpecName "kube-api-access-62dtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.918837 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.918341    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"]
Oct 30 19:49:19.923911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.923374    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"]
Oct 30 19:49:19.937129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.937100    6777 generic.go:356] "Generic (PLEG): container finished" podID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerID="10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe" exitCode=0
Oct 30 19:49:19.937281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.937188    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" event={"ID":"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce","Type":"ContainerDied","Data":"10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe"}
Oct 30 19:49:19.937281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.937205    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"
Oct 30 19:49:19.937281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.937267    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw" event={"ID":"4eb6c1cf-e34a-49af-b7e6-9080d5c699ce","Type":"ContainerDied","Data":"e496a72886c17afb2885e9c5ee6f0ccd1b47608cf35adc3a6178af01217f186d"}
Oct 30 19:49:19.937464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.937300    6777 scope.go:122] "RemoveContainer" containerID="10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe"
Oct 30 19:49:19.939106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.939078    6777 generic.go:356] "Generic (PLEG): container finished" podID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerID="5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9" exitCode=0
Oct 30 19:49:19.939226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.939160    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"
Oct 30 19:49:19.939226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.939178    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" event={"ID":"1a01e0ca-b6ee-44c6-be57-f94a25d39f21","Type":"ContainerDied","Data":"5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9"}
Oct 30 19:49:19.939363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.939256    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58" event={"ID":"1a01e0ca-b6ee-44c6-be57-f94a25d39f21","Type":"ContainerDied","Data":"25e62c6f99ff2372a2447cdd852da385ee9b67830835dbf417585982979e861e"}
Oct 30 19:49:19.943599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.943577    6777 generic.go:356] "Generic (PLEG): container finished" podID="ffc394be-a643-4647-9999-d163fc1a2fad" containerID="b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec" exitCode=0
Oct 30 19:49:19.943718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.943641    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"
Oct 30 19:49:19.943718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.943659    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" event={"ID":"ffc394be-a643-4647-9999-d163fc1a2fad","Type":"ContainerDied","Data":"b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec"}
Oct 30 19:49:19.943838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.943719    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p" event={"ID":"ffc394be-a643-4647-9999-d163fc1a2fad","Type":"ContainerDied","Data":"fccfc2abaeb13a03bc5e72e5d192eb00f42695a5bac14490bef6e70e9451faac"}
Oct 30 19:49:19.945165 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.945144    6777 generic.go:356] "Generic (PLEG): container finished" podID="4405b1c4-1a26-4592-a367-e763601cba0c" containerID="028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8" exitCode=0
Oct 30 19:49:19.945281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.945181    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" event={"ID":"4405b1c4-1a26-4592-a367-e763601cba0c","Type":"ContainerDied","Data":"028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8"}
Oct 30 19:49:19.945281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.945196    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"
Oct 30 19:49:19.945281 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.945215    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl" event={"ID":"4405b1c4-1a26-4592-a367-e763601cba0c","Type":"ContainerDied","Data":"02c72e35df85fc1f0889c74625e41bd91f77967305aeaafff5e996e4f5c5e4f2"}
Oct 30 19:49:19.954976 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.954955    6777 scope.go:122] "RemoveContainer" containerID="10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe"
Oct 30 19:49:19.955446 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:19.955332    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe\": container with ID starting with 10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe not found: ID does not exist" containerID="10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe"
Oct 30 19:49:19.955446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.955365    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe"} err="failed to get container status \"10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe\": rpc error: code = NotFound desc = could not find container \"10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe\": container with ID starting with 10788f7acca1564a447797911d97e3d4c19c32ff57e5f7b64c9abf3e9e0ddabe not found: ID does not exist"
Oct 30 19:49:19.955446 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.955390    6777 scope.go:122] "RemoveContainer" containerID="5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9"
Oct 30 19:49:19.966541 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.966524    6777 scope.go:122] "RemoveContainer" containerID="b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b"
Oct 30 19:49:19.977514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.977494    6777 scope.go:122] "RemoveContainer" containerID="5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9"
Oct 30 19:49:19.977822 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:19.977800    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9\": container with ID starting with 5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9 not found: ID does not exist" containerID="5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9"
Oct 30 19:49:19.977914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.977833    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9"} err="failed to get container status \"5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9\": rpc error: code = NotFound desc = could not find container \"5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9\": container with ID starting with 5d56f4654244c172fa13a28fa5bbc166744460c711b013c0de197d95e559d1c9 not found: ID does not exist"
Oct 30 19:49:19.977914 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.977856    6777 scope.go:122] "RemoveContainer" containerID="b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b"
Oct 30 19:49:19.978975 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:19.978951    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b\": container with ID starting with b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b not found: ID does not exist" containerID="b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b"
Oct 30 19:49:19.979125 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.978987    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b"} err="failed to get container status \"b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b\": rpc error: code = NotFound desc = could not find container \"b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b\": container with ID starting with b0da1bb7aae1834b65e7659be21883aa60a014b07cbe5f2f307f7bac34e70b6b not found: ID does not exist"
Oct 30 19:49:19.979125 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.979007    6777 scope.go:122] "RemoveContainer" containerID="b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec"
Oct 30 19:49:19.988665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.988650    6777 scope.go:122] "RemoveContainer" containerID="b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec"
Oct 30 19:49:19.988932 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:19.988912    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec\": container with ID starting with b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec not found: ID does not exist" containerID="b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec"
Oct 30 19:49:19.988980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.988942    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec"} err="failed to get container status \"b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec\": rpc error: code = NotFound desc = could not find container \"b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec\": container with ID starting with b4ab5e3f7eb19bfbceba6c935878d8c698c7c4e2a620cc1218759a36d42afaec not found: ID does not exist"
Oct 30 19:49:19.988980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.988957    6777 scope.go:122] "RemoveContainer" containerID="028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8"
Oct 30 19:49:19.995698 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:19.995675    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"]
Oct 30 19:49:20.000192 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.000172    6777 scope.go:122] "RemoveContainer" containerID="028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8"
Oct 30 19:49:20.000449 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.000431    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-xb7mw"]
Oct 30 19:49:20.000523 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:20.000455    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8\": container with ID starting with 028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8 not found: ID does not exist" containerID="028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8"
Oct 30 19:49:20.000523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.000476    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8"} err="failed to get container status \"028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8\": rpc error: code = NotFound desc = could not find container \"028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8\": container with ID starting with 028a1f8214720031c1676322e7ff1fba0a48a219e8822c2211072856d9392df8 not found: ID does not exist"
Oct 30 19:49:20.011997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.011975    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"]
Oct 30 19:49:20.012254 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012236    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-socket\") pod \"ffc394be-a643-4647-9999-d163fc1a2fad\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") "
Oct 30 19:49:20.012305 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012276    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-kube-api-access-trnjm\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-kube-api-access-trnjm\") pod \"ffc394be-a643-4647-9999-d163fc1a2fad\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") "
Oct 30 19:49:20.012344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012312    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffc394be-a643-4647-9999-d163fc1a2fad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc394be-a643-4647-9999-d163fc1a2fad-istio-podinfo\") pod \"ffc394be-a643-4647-9999-d163fc1a2fad\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") "
Oct 30 19:49:20.012398 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012357    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-certs\") pod \"ffc394be-a643-4647-9999-d163fc1a2fad\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") "
Oct 30 19:49:20.012458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012396    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-data\") pod \"ffc394be-a643-4647-9999-d163fc1a2fad\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") "
Oct 30 19:49:20.012458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012434    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-istio-token\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-istio-token\") pod \"ffc394be-a643-4647-9999-d163fc1a2fad\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") "
Oct 30 19:49:20.012569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012506    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-credential-socket\") pod \"ffc394be-a643-4647-9999-d163fc1a2fad\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") "
Oct 30 19:49:20.012569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012518    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-socket" pod "ffc394be-a643-4647-9999-d163fc1a2fad" (UID: "ffc394be-a643-4647-9999-d163fc1a2fad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:20.012569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012536    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc394be-a643-4647-9999-d163fc1a2fad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc394be-a643-4647-9999-d163fc1a2fad-istiod-ca-cert\") pod \"ffc394be-a643-4647-9999-d163fc1a2fad\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") "
Oct 30 19:49:20.012732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012589    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-envoy\") pod \"ffc394be-a643-4647-9999-d163fc1a2fad\" (UID: \"ffc394be-a643-4647-9999-d163fc1a2fad\") "
Oct 30 19:49:20.012799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012721    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-data" pod "ffc394be-a643-4647-9999-d163fc1a2fad" (UID: "ffc394be-a643-4647-9999-d163fc1a2fad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:20.012799 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012749    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-certs" pod "ffc394be-a643-4647-9999-d163fc1a2fad" (UID: "ffc394be-a643-4647-9999-d163fc1a2fad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:20.012995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.012966    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-credential-socket" pod "ffc394be-a643-4647-9999-d163fc1a2fad" (UID: "ffc394be-a643-4647-9999-d163fc1a2fad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:20.013122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013082    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc394be-a643-4647-9999-d163fc1a2fad-istiod-ca-cert" pod "ffc394be-a643-4647-9999-d163fc1a2fad" (UID: "ffc394be-a643-4647-9999-d163fc1a2fad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:20.013191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013160    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013183    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013200    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6qrl\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-kube-api-access-w6qrl\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013217    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013230    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013244    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013257    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62dtt\" (UniqueName: \"kubernetes.io/projected/4405b1c4-1a26-4592-a367-e763601cba0c-kube-api-access-62dtt\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013272    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013286    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013301    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013315    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013328    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013342    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013356    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013369    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013383    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013396    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4405b1c4-1a26-4592-a367-e763601cba0c-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013409    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013424    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013438    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013453    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013467    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013482    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013496    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4405b1c4-1a26-4592-a367-e763601cba0c-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013509    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013522    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4405b1c4-1a26-4592-a367-e763601cba0c-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013537    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013552    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.013572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.013565    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhw4v\" (UniqueName: \"kubernetes.io/projected/1a01e0ca-b6ee-44c6-be57-f94a25d39f21-kube-api-access-rhw4v\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.015553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.015511    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-envoy" pod "ffc394be-a643-4647-9999-d163fc1a2fad" (UID: "ffc394be-a643-4647-9999-d163fc1a2fad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:20.015772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.015748    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-kube-api-access-trnjm" pod "ffc394be-a643-4647-9999-d163fc1a2fad" (UID: "ffc394be-a643-4647-9999-d163fc1a2fad"). InnerVolumeSpecName "kube-api-access-trnjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:20.015891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.015873    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-istio-token" pod "ffc394be-a643-4647-9999-d163fc1a2fad" (UID: "ffc394be-a643-4647-9999-d163fc1a2fad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:20.016054 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.016041    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffc394be-a643-4647-9999-d163fc1a2fad-istio-podinfo" pod "ffc394be-a643-4647-9999-d163fc1a2fad" (UID: "ffc394be-a643-4647-9999-d163fc1a2fad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:20.017383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.017364    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-crtpl"]
Oct 30 19:49:20.036905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.036874    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"]
Oct 30 19:49:20.043345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.043318    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-l4g58"]
Oct 30 19:49:20.112321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.112284    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" path="/var/lib/kubelet/pods/1a01e0ca-b6ee-44c6-be57-f94a25d39f21/volumes"
Oct 30 19:49:20.112781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.112767    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" path="/var/lib/kubelet/pods/4405b1c4-1a26-4592-a367-e763601cba0c/volumes"
Oct 30 19:49:20.113200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.113187    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" path="/var/lib/kubelet/pods/4eb6c1cf-e34a-49af-b7e6-9080d5c699ce/volumes"
Oct 30 19:49:20.114361 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.114339    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc394be-a643-4647-9999-d163fc1a2fad-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.114473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.114364    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc394be-a643-4647-9999-d163fc1a2fad-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.114473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.114379    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trnjm\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-kube-api-access-trnjm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.114473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.114393    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc394be-a643-4647-9999-d163fc1a2fad-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.114473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.114406    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc394be-a643-4647-9999-d163fc1a2fad-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.261575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.261489    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"]
Oct 30 19:49:20.267829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:20.267800    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-5j29p"]
Oct 30 19:49:22.112229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.112191    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" path="/var/lib/kubelet/pods/ffc394be-a643-4647-9999-d163fc1a2fad/volumes"
Oct 30 19:49:22.282006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.281967    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"]
Oct 30 19:49:22.282242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.282216    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy" containerID="cri-o://24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee" gracePeriod=30
Oct 30 19:49:22.523270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.523241    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:49:22.644002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.643964    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52606f4d-8ec1-42d9-b49a-67218a118f6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606f4d-8ec1-42d9-b49a-67218a118f6f-plugins-config-volume\") pod \"52606f4d-8ec1-42d9-b49a-67218a118f6f\" (UID: \"52606f4d-8ec1-42d9-b49a-67218a118f6f\") "
Oct 30 19:49:22.644251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.644103    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52606f4d-8ec1-42d9-b49a-67218a118f6f-kube-api-access-pqt88\" (UniqueName: \"kubernetes.io/projected/52606f4d-8ec1-42d9-b49a-67218a118f6f-kube-api-access-pqt88\") pod \"52606f4d-8ec1-42d9-b49a-67218a118f6f\" (UID: \"52606f4d-8ec1-42d9-b49a-67218a118f6f\") "
Oct 30 19:49:22.644466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.644438    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52606f4d-8ec1-42d9-b49a-67218a118f6f-plugins-config-volume" pod "52606f4d-8ec1-42d9-b49a-67218a118f6f" (UID: "52606f4d-8ec1-42d9-b49a-67218a118f6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:22.646537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.646510    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52606f4d-8ec1-42d9-b49a-67218a118f6f-kube-api-access-pqt88" pod "52606f4d-8ec1-42d9-b49a-67218a118f6f" (UID: "52606f4d-8ec1-42d9-b49a-67218a118f6f"). InnerVolumeSpecName "kube-api-access-pqt88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:22.746149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.746002    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52606f4d-8ec1-42d9-b49a-67218a118f6f-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:22.746149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.746043    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqt88\" (UniqueName: \"kubernetes.io/projected/52606f4d-8ec1-42d9-b49a-67218a118f6f-kube-api-access-pqt88\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:22.968204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.968160    6777 generic.go:356] "Generic (PLEG): container finished" podID="52606f4d-8ec1-42d9-b49a-67218a118f6f" containerID="e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b" exitCode=0
Oct 30 19:49:22.968387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.968221    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"
Oct 30 19:49:22.968387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.968242    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6" event={"ID":"52606f4d-8ec1-42d9-b49a-67218a118f6f","Type":"ContainerDied","Data":"e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b"}
Oct 30 19:49:22.968387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.968334    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6" event={"ID":"52606f4d-8ec1-42d9-b49a-67218a118f6f","Type":"ContainerDied","Data":"7e4229ddf26c4a997d92e5636074336ca66cb176f1b72b3dec079213c4238943"}
Oct 30 19:49:22.968387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.968362    6777 scope.go:122] "RemoveContainer" containerID="e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b"
Oct 30 19:49:22.978660 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.978579    6777 scope.go:122] "RemoveContainer" containerID="e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b"
Oct 30 19:49:22.979217 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:22.979191    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b\": container with ID starting with e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b not found: ID does not exist" containerID="e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b"
Oct 30 19:49:22.979321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.979225    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b"} err="failed to get container status \"e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b\": rpc error: code = NotFound desc = could not find container \"e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b\": container with ID starting with e446b0286d333430fdb61b7ba8afa9fd8c144cfec2ea7b86cbf23c346811976b not found: ID does not exist"
Oct 30 19:49:22.993896 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.993864    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"]
Oct 30 19:49:22.997930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:22.997859    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39b867-6bb6768bb4-9n2s6"]
Oct 30 19:49:24.111731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:24.111694    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52606f4d-8ec1-42d9-b49a-67218a118f6f" path="/var/lib/kubelet/pods/52606f4d-8ec1-42d9-b49a-67218a118f6f/volumes"
Oct 30 19:49:26.210080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210028    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"]
Oct 30 19:49:26.210493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210477    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy"
Oct 30 19:49:26.210534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210494    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy"
Oct 30 19:49:26.210534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210508    6777 cpu_manager.go:414] "containerMap: removing container" podUID="52606f4d-8ec1-42d9-b49a-67218a118f6f" containerName="payload-processing"
Oct 30 19:49:26.210534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210514    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52606f4d-8ec1-42d9-b49a-67218a118f6f" containerName="payload-processing"
Oct 30 19:49:26.210534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210526    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e67fd308-ff22-476f-82a1-e22d41c8381e" containerName="payload-processing"
Oct 30 19:49:26.210534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210531    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67fd308-ff22-476f-82a1-e22d41c8381e" containerName="payload-processing"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210538    6777 cpu_manager.go:414] "containerMap: removing container" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210542    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210553    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7ffe77ce-28ce-4512-a13b-f46175c14dc4" containerName="revoke-keys"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210557    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ffe77ce-28ce-4512-a13b-f46175c14dc4" containerName="revoke-keys"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210573    6777 cpu_manager.go:414] "containerMap: removing container" podUID="095c2e53-0ad0-40f6-af86-0bad69490983" containerName="payload-pre-processing"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210579    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095c2e53-0ad0-40f6-af86-0bad69490983" containerName="payload-pre-processing"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210593    6777 cpu_manager.go:414] "containerMap: removing container" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210598    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210611    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" containerName="istio-proxy"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210616    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" containerName="istio-proxy"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210675    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210683    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="52606f4d-8ec1-42d9-b49a-67218a118f6f" containerName="payload-processing"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210690    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67fd308-ff22-476f-82a1-e22d41c8381e" containerName="payload-processing"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210695    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb6c1cf-e34a-49af-b7e6-9080d5c699ce" containerName="istio-proxy"
Oct 30 19:49:26.210697 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210701    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ffe77ce-28ce-4512-a13b-f46175c14dc4" containerName="revoke-keys"
Oct 30 19:49:26.211189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210709    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="095c2e53-0ad0-40f6-af86-0bad69490983" containerName="payload-pre-processing"
Oct 30 19:49:26.211189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210716    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="4405b1c4-1a26-4592-a367-e763601cba0c" containerName="istio-proxy"
Oct 30 19:49:26.211189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.210723    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc394be-a643-4647-9999-d163fc1a2fad" containerName="istio-proxy"
Oct 30 19:49:26.213910 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.213882    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.227556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.223533    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"]
Oct 30 19:49:26.314345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.314304    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:26.314546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.314400    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:49:26.314546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.314401    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:49:26.314546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.314401    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:49:26.387424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.387376    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7f78\" (UniqueName: \"kubernetes.io/projected/d9bbcbd7-69f3-402c-8206-2c050904b75d-kube-api-access-m7f78\") pod \"maas-api-revoke-sub-255d55c5-f5trd\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.387593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.387439    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-255d55c5-f5trd\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.387593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.387545    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-255d55c5-f5trd\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.489319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.489222    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-255d55c5-f5trd\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.489319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.489282    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-255d55c5-f5trd\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.489565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.489389    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7f78\" (UniqueName: \"kubernetes.io/projected/d9bbcbd7-69f3-402c-8206-2c050904b75d-kube-api-access-m7f78\") pod \"maas-api-revoke-sub-255d55c5-f5trd\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.489928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.489902    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-255d55c5-f5trd\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.491709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.491685    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-255d55c5-f5trd\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.502474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.502439    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7f78\" (UniqueName: \"kubernetes.io/projected/d9bbcbd7-69f3-402c-8206-2c050904b75d-kube-api-access-m7f78\") pod \"maas-api-revoke-sub-255d55c5-f5trd\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.535294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.535255    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:26.814395 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.814367    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"]
Oct 30 19:49:26.816812 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:49:26.816786    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bbcbd7_69f3_402c_8206_2c050904b75d.slice/crio-c5bf46b595cca5d6fe19c1127db53acb8661411985f2e63002ae2f06b03f5a5e WatchSource:0}: Error finding container c5bf46b595cca5d6fe19c1127db53acb8661411985f2e63002ae2f06b03f5a5e: Status 404 returned error can't find the container with id c5bf46b595cca5d6fe19c1127db53acb8661411985f2e63002ae2f06b03f5a5e
Oct 30 19:49:26.996282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:26.996245    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd" event={"ID":"d9bbcbd7-69f3-402c-8206-2c050904b75d","Type":"ContainerStarted","Data":"c5bf46b595cca5d6fe19c1127db53acb8661411985f2e63002ae2f06b03f5a5e"}
Oct 30 19:49:27.553950 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.553919    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:49:27.702367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702332    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-credential-socket\") pod \"e7489c99-67e2-438e-9593-3f552755ff5c\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") "
Oct 30 19:49:27.702574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702373    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-envoy\") pod \"e7489c99-67e2-438e-9593-3f552755ff5c\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") "
Oct 30 19:49:27.702574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702405    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-data\") pod \"e7489c99-67e2-438e-9593-3f552755ff5c\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") "
Oct 30 19:49:27.702574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702459    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-kube-api-access-w7z47\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-kube-api-access-w7z47\") pod \"e7489c99-67e2-438e-9593-3f552755ff5c\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") "
Oct 30 19:49:27.702574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702492    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7489c99-67e2-438e-9593-3f552755ff5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7489c99-67e2-438e-9593-3f552755ff5c-istiod-ca-cert\") pod \"e7489c99-67e2-438e-9593-3f552755ff5c\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") "
Oct 30 19:49:27.702809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702609    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7489c99-67e2-438e-9593-3f552755ff5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7489c99-67e2-438e-9593-3f552755ff5c-istio-podinfo\") pod \"e7489c99-67e2-438e-9593-3f552755ff5c\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") "
Oct 30 19:49:27.702809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702653    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-certs\") pod \"e7489c99-67e2-438e-9593-3f552755ff5c\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") "
Oct 30 19:49:27.702809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702713    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-credential-socket" pod "e7489c99-67e2-438e-9593-3f552755ff5c" (UID: "e7489c99-67e2-438e-9593-3f552755ff5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.702809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702728    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-istio-token\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-istio-token\") pod \"e7489c99-67e2-438e-9593-3f552755ff5c\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") "
Oct 30 19:49:27.702809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702764    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-socket\") pod \"e7489c99-67e2-438e-9593-3f552755ff5c\" (UID: \"e7489c99-67e2-438e-9593-3f552755ff5c\") "
Oct 30 19:49:27.702809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702796    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-data" pod "e7489c99-67e2-438e-9593-3f552755ff5c" (UID: "e7489c99-67e2-438e-9593-3f552755ff5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.703179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702871    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7489c99-67e2-438e-9593-3f552755ff5c-istiod-ca-cert" pod "e7489c99-67e2-438e-9593-3f552755ff5c" (UID: "e7489c99-67e2-438e-9593-3f552755ff5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:27.703179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.702952    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-certs" pod "e7489c99-67e2-438e-9593-3f552755ff5c" (UID: "e7489c99-67e2-438e-9593-3f552755ff5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.703179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.703166    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-socket" pod "e7489c99-67e2-438e-9593-3f552755ff5c" (UID: "e7489c99-67e2-438e-9593-3f552755ff5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.703349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.703268    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.703349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.703284    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.703349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.703293    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.703349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.703303    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7489c99-67e2-438e-9593-3f552755ff5c-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.704911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.704880    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-envoy" pod "e7489c99-67e2-438e-9593-3f552755ff5c" (UID: "e7489c99-67e2-438e-9593-3f552755ff5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.705051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.704935    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7489c99-67e2-438e-9593-3f552755ff5c-istio-podinfo" pod "e7489c99-67e2-438e-9593-3f552755ff5c" (UID: "e7489c99-67e2-438e-9593-3f552755ff5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:27.705051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.704974    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-istio-token" pod "e7489c99-67e2-438e-9593-3f552755ff5c" (UID: "e7489c99-67e2-438e-9593-3f552755ff5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:27.705051 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.704993    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-kube-api-access-w7z47" pod "e7489c99-67e2-438e-9593-3f552755ff5c" (UID: "e7489c99-67e2-438e-9593-3f552755ff5c"). InnerVolumeSpecName "kube-api-access-w7z47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:27.803974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.803927    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7489c99-67e2-438e-9593-3f552755ff5c-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.803974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.803966    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.803974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.803976    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.803974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.803984    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7489c99-67e2-438e-9593-3f552755ff5c-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.804303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:27.803994    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7z47\" (UniqueName: \"kubernetes.io/projected/e7489c99-67e2-438e-9593-3f552755ff5c-kube-api-access-w7z47\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:28.003489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.003395    6777 generic.go:356] "Generic (PLEG): container finished" podID="e7489c99-67e2-438e-9593-3f552755ff5c" containerID="24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee" exitCode=0
Oct 30 19:49:28.003489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.003470    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" event={"ID":"e7489c99-67e2-438e-9593-3f552755ff5c","Type":"ContainerDied","Data":"24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee"}
Oct 30 19:49:28.003775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.003479    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"
Oct 30 19:49:28.003775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.003567    6777 scope.go:122] "RemoveContainer" containerID="24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee"
Oct 30 19:49:28.003775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.003553    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr" event={"ID":"e7489c99-67e2-438e-9593-3f552755ff5c","Type":"ContainerDied","Data":"f2119458459d0cb42eb3463c16920a4a5f4b63b4a71bc35a5e9c1133cde2a9bd"}
Oct 30 19:49:28.005428 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.005398    6777 generic.go:356] "Generic (PLEG): container finished" podID="d9bbcbd7-69f3-402c-8206-2c050904b75d" containerID="decbc999e7e0dc4cbe789f27ef4e3d6637d4fb24ce2d7e405f59769ca1335c61" exitCode=0
Oct 30 19:49:28.005568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.005499    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd" event={"ID":"d9bbcbd7-69f3-402c-8206-2c050904b75d","Type":"ContainerDied","Data":"decbc999e7e0dc4cbe789f27ef4e3d6637d4fb24ce2d7e405f59769ca1335c61"}
Oct 30 19:49:28.014568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.014550    6777 scope.go:122] "RemoveContainer" containerID="24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee"
Oct 30 19:49:28.014887 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:28.014868    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee\": container with ID starting with 24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee not found: ID does not exist" containerID="24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee"
Oct 30 19:49:28.014958 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.014900    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee"} err="failed to get container status \"24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee\": rpc error: code = NotFound desc = could not find container \"24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee\": container with ID starting with 24ac6ecf64f136617bf56b736ea063a49172f35b5b2aea99843097e7247041ee not found: ID does not exist"
Oct 30 19:49:28.042007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.041781    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"]
Oct 30 19:49:28.047548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.047516    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-ct8nr"]
Oct 30 19:49:28.111559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.111521    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" path="/var/lib/kubelet/pods/e7489c99-67e2-438e-9593-3f552755ff5c/volumes"
Oct 30 19:49:28.653577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:28.653535    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn" podUID="ba70864a-66bd-44b6-9923-0ac9f00260fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:49:29.157345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.157318    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:29.318868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.318828    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bbcbd7-69f3-402c-8206-2c050904b75d-kube-api-access-m7f78\" (UniqueName: \"kubernetes.io/projected/d9bbcbd7-69f3-402c-8206-2c050904b75d-kube-api-access-m7f78\") pod \"d9bbcbd7-69f3-402c-8206-2c050904b75d\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") "
Oct 30 19:49:29.319136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.318898    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-openshift-ca\") pod \"d9bbcbd7-69f3-402c-8206-2c050904b75d\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") "
Oct 30 19:49:29.319136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.318991    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-xks-ca\") pod \"d9bbcbd7-69f3-402c-8206-2c050904b75d\" (UID: \"d9bbcbd7-69f3-402c-8206-2c050904b75d\") "
Oct 30 19:49:29.319366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.319335    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-openshift-ca" pod "d9bbcbd7-69f3-402c-8206-2c050904b75d" (UID: "d9bbcbd7-69f3-402c-8206-2c050904b75d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:29.319486 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.319467    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:29.321101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.321076    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-xks-ca" pod "d9bbcbd7-69f3-402c-8206-2c050904b75d" (UID: "d9bbcbd7-69f3-402c-8206-2c050904b75d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:29.321170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.321151    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bbcbd7-69f3-402c-8206-2c050904b75d-kube-api-access-m7f78" pod "d9bbcbd7-69f3-402c-8206-2c050904b75d" (UID: "d9bbcbd7-69f3-402c-8206-2c050904b75d"). InnerVolumeSpecName "kube-api-access-m7f78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:29.420627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.420589    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7f78\" (UniqueName: \"kubernetes.io/projected/d9bbcbd7-69f3-402c-8206-2c050904b75d-kube-api-access-m7f78\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:29.420627 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.420621    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bbcbd7-69f3-402c-8206-2c050904b75d-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:29.785277 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.785244    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:49:29.925543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.925453    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba70864a-66bd-44b6-9923-0ac9f00260fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba70864a-66bd-44b6-9923-0ac9f00260fd-plugins-config-volume\") pod \"ba70864a-66bd-44b6-9923-0ac9f00260fd\" (UID: \"ba70864a-66bd-44b6-9923-0ac9f00260fd\") "
Oct 30 19:49:29.925739 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.925553    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba70864a-66bd-44b6-9923-0ac9f00260fd-kube-api-access-k7drw\" (UniqueName: \"kubernetes.io/projected/ba70864a-66bd-44b6-9923-0ac9f00260fd-kube-api-access-k7drw\") pod \"ba70864a-66bd-44b6-9923-0ac9f00260fd\" (UID: \"ba70864a-66bd-44b6-9923-0ac9f00260fd\") "
Oct 30 19:49:29.925868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.925842    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba70864a-66bd-44b6-9923-0ac9f00260fd-plugins-config-volume" pod "ba70864a-66bd-44b6-9923-0ac9f00260fd" (UID: "ba70864a-66bd-44b6-9923-0ac9f00260fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:29.927671 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:29.927648    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba70864a-66bd-44b6-9923-0ac9f00260fd-kube-api-access-k7drw" pod "ba70864a-66bd-44b6-9923-0ac9f00260fd" (UID: "ba70864a-66bd-44b6-9923-0ac9f00260fd"). InnerVolumeSpecName "kube-api-access-k7drw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:30.020221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.020191    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"
Oct 30 19:49:30.020414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.020190    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd" event={"ID":"d9bbcbd7-69f3-402c-8206-2c050904b75d","Type":"ContainerDied","Data":"c5bf46b595cca5d6fe19c1127db53acb8661411985f2e63002ae2f06b03f5a5e"}
Oct 30 19:49:30.020414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.020334    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5bf46b595cca5d6fe19c1127db53acb8661411985f2e63002ae2f06b03f5a5e"
Oct 30 19:49:30.021641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.021615    6777 generic.go:356] "Generic (PLEG): container finished" podID="ba70864a-66bd-44b6-9923-0ac9f00260fd" containerID="7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f" exitCode=0
Oct 30 19:49:30.021795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.021657    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"
Oct 30 19:49:30.021795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.021694    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn" event={"ID":"ba70864a-66bd-44b6-9923-0ac9f00260fd","Type":"ContainerDied","Data":"7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f"}
Oct 30 19:49:30.021795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.021755    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn" event={"ID":"ba70864a-66bd-44b6-9923-0ac9f00260fd","Type":"ContainerDied","Data":"4dc8be2fe868d5df2bcca81e64d593a102ffba9ea8a4447a96c5f371ec0648a3"}
Oct 30 19:49:30.021795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.021778    6777 scope.go:122] "RemoveContainer" containerID="7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f"
Oct 30 19:49:30.026573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.026547    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba70864a-66bd-44b6-9923-0ac9f00260fd-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.026573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.026570    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7drw\" (UniqueName: \"kubernetes.io/projected/ba70864a-66bd-44b6-9923-0ac9f00260fd-kube-api-access-k7drw\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.032803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.032786    6777 scope.go:122] "RemoveContainer" containerID="7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f"
Oct 30 19:49:30.033133 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:30.033110    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f\": container with ID starting with 7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f not found: ID does not exist" containerID="7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f"
Oct 30 19:49:30.033195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.033142    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f"} err="failed to get container status \"7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f\": rpc error: code = NotFound desc = could not find container \"7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f\": container with ID starting with 7b48fb9a0d1a4cd64500303e6afbc6b99c29a1663686049a883ec22d3b153b6f not found: ID does not exist"
Oct 30 19:49:30.050256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.050224    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"]
Oct 30 19:49:30.054812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.054782    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fd7c77-5cdd779744-f5kzn"]
Oct 30 19:49:30.111745 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:30.111711    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba70864a-66bd-44b6-9923-0ac9f00260fd" path="/var/lib/kubelet/pods/ba70864a-66bd-44b6-9923-0ac9f00260fd/volumes"
Oct 30 19:49:34.949392 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:34.949344    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" containerID="cri-o://a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f" gracePeriod=45
Oct 30 19:49:34.951759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:34.951720    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main" containerID="cri-o://ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f" gracePeriod=45
Oct 30 19:49:35.220674 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.220646    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:49:35.223768 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.223743    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:49:35.276955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.276912    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-home\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-home\") pod \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") "
Oct 30 19:49:35.277221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.276968    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-tmp-dir\") pod \"3ff91e8e-7ada-4741-815e-06c9955b4151\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") "
Oct 30 19:49:35.277221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277019    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff91e8e-7ada-4741-815e-06c9955b4151-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff91e8e-7ada-4741-815e-06c9955b4151-tls-certs\") pod \"3ff91e8e-7ada-4741-815e-06c9955b4151\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") "
Oct 30 19:49:35.277221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277084    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-dshm\") pod \"3ff91e8e-7ada-4741-815e-06c9955b4151\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") "
Oct 30 19:49:35.277221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277109    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-home\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-home\") pod \"3ff91e8e-7ada-4741-815e-06c9955b4151\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") "
Oct 30 19:49:35.277221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277145    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-model-cache\") pod \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") "
Oct 30 19:49:35.277515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277231    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-model-cache\") pod \"3ff91e8e-7ada-4741-815e-06c9955b4151\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") "
Oct 30 19:49:35.277515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277263    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tls-certs\") pod \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") "
Oct 30 19:49:35.277515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277316    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-kube-api-access-vmxb4\" (UniqueName: \"kubernetes.io/projected/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-kube-api-access-vmxb4\") pod \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") "
Oct 30 19:49:35.277515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277396    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tmp-dir\") pod \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") "
Oct 30 19:49:35.277515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277432    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-dshm\") pod \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\" (UID: \"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a\") "
Oct 30 19:49:35.277515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.277463    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff91e8e-7ada-4741-815e-06c9955b4151-kube-api-access-x95t5\" (UniqueName: \"kubernetes.io/projected/3ff91e8e-7ada-4741-815e-06c9955b4151-kube-api-access-x95t5\") pod \"3ff91e8e-7ada-4741-815e-06c9955b4151\" (UID: \"3ff91e8e-7ada-4741-815e-06c9955b4151\") "
Oct 30 19:49:35.278564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.278375    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tmp-dir" pod "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" (UID: "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:35.278564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.278381    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-model-cache" pod "3ff91e8e-7ada-4741-815e-06c9955b4151" (UID: "3ff91e8e-7ada-4741-815e-06c9955b4151"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:35.278750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.278588    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-home" pod "3ff91e8e-7ada-4741-815e-06c9955b4151" (UID: "3ff91e8e-7ada-4741-815e-06c9955b4151"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:35.278750 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.278651    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-model-cache" pod "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" (UID: "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:35.278966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.278862    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-tmp-dir" pod "3ff91e8e-7ada-4741-815e-06c9955b4151" (UID: "3ff91e8e-7ada-4741-815e-06c9955b4151"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:35.278966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.278933    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-home" pod "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" (UID: "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:35.282600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.282566    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff91e8e-7ada-4741-815e-06c9955b4151-kube-api-access-x95t5" pod "3ff91e8e-7ada-4741-815e-06c9955b4151" (UID: "3ff91e8e-7ada-4741-815e-06c9955b4151"). InnerVolumeSpecName "kube-api-access-x95t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:35.282822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.282797    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-kube-api-access-vmxb4" pod "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" (UID: "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a"). InnerVolumeSpecName "kube-api-access-vmxb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:35.283340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.283315    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-dshm" pod "3ff91e8e-7ada-4741-815e-06c9955b4151" (UID: "3ff91e8e-7ada-4741-815e-06c9955b4151"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:35.283442 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.283402    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff91e8e-7ada-4741-815e-06c9955b4151-tls-certs" pod "3ff91e8e-7ada-4741-815e-06c9955b4151" (UID: "3ff91e8e-7ada-4741-815e-06c9955b4151"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:35.283526 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.283509    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tls-certs" pod "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" (UID: "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:35.285312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.285267    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-dshm" pod "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" (UID: "cfe94c2e-56f0-4979-8b7c-d83baf87ea6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:35.378755 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378719    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmxb4\" (UniqueName: \"kubernetes.io/projected/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-kube-api-access-vmxb4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378773    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378783    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378793    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x95t5\" (UniqueName: \"kubernetes.io/projected/3ff91e8e-7ada-4741-815e-06c9955b4151-kube-api-access-x95t5\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378802    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378810    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378817    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff91e8e-7ada-4741-815e-06c9955b4151-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378826    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378833    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378841    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378849    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff91e8e-7ada-4741-815e-06c9955b4151-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:35.378919 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:35.378857    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:36.057470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.057434    6777 generic.go:356] "Generic (PLEG): container finished" podID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerID="ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f" exitCode=0
Oct 30 19:49:36.057470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.057474    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"
Oct 30 19:49:36.058001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.057515    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" event={"ID":"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a","Type":"ContainerDied","Data":"ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f"}
Oct 30 19:49:36.058001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.057562    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl" event={"ID":"cfe94c2e-56f0-4979-8b7c-d83baf87ea6a","Type":"ContainerDied","Data":"b0bf33442d9293d6908554214693996a2f6cda50f2995d28fc3064ebddd5a2a6"}
Oct 30 19:49:36.058001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.057583    6777 scope.go:122] "RemoveContainer" containerID="ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f"
Oct 30 19:49:36.059184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.059159    6777 generic.go:356] "Generic (PLEG): container finished" podID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerID="a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f" exitCode=0
Oct 30 19:49:36.059284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.059201    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"
Oct 30 19:49:36.059284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.059225    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" event={"ID":"3ff91e8e-7ada-4741-815e-06c9955b4151","Type":"ContainerDied","Data":"a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f"}
Oct 30 19:49:36.059284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.059257    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" event={"ID":"3ff91e8e-7ada-4741-815e-06c9955b4151","Type":"ContainerDied","Data":"fcfcf69b553be810ccef5f714ea2d5ebd14b7a8aa9fe3ff42e81506f4bcb4b2d"}
Oct 30 19:49:36.068221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.068137    6777 scope.go:122] "RemoveContainer" containerID="ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f"
Oct 30 19:49:36.068534 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:36.068507    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f\": container with ID starting with ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f not found: ID does not exist" containerID="ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f"
Oct 30 19:49:36.068628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.068544    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f"} err="failed to get container status \"ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f\": rpc error: code = NotFound desc = could not find container \"ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f\": container with ID starting with ebe47f301afdc7aff3c6d17dbdcc55fb253bf96efdd00fb74ce659c26e0b720f not found: ID does not exist"
Oct 30 19:49:36.068628 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.068567    6777 scope.go:122] "RemoveContainer" containerID="a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f"
Oct 30 19:49:36.077835 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.077816    6777 scope.go:122] "RemoveContainer" containerID="a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f"
Oct 30 19:49:36.078118 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:36.078095    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f\": container with ID starting with a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f not found: ID does not exist" containerID="a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f"
Oct 30 19:49:36.078181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.078127    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f"} err="failed to get container status \"a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f\": rpc error: code = NotFound desc = could not find container \"a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f\": container with ID starting with a56f085e99f313de8a6e1369280b5db07a2288289dac4c3be478221abd4f959f not found: ID does not exist"
Oct 30 19:49:36.084217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.084190    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"]
Oct 30 19:49:36.088903 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.088873    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/simulator-w3-17e904-kserve-6688676f9b-ld4zl"]
Oct 30 19:49:36.100867 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.100837    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"]
Oct 30 19:49:36.105022 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.104996    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf"]
Oct 30 19:49:36.111307 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.111276    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" path="/var/lib/kubelet/pods/3ff91e8e-7ada-4741-815e-06c9955b4151/volumes"
Oct 30 19:49:36.111796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.111777    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" path="/var/lib/kubelet/pods/cfe94c2e-56f0-4979-8b7c-d83baf87ea6a/volumes"
Oct 30 19:49:36.215529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:36.215423    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17e904/premium-w3-17e904-kserve-5cf57bbb69-77lzf" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:49:37.886527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:37.886485    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:49:43.263622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:43.263534    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:49:43.639708 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:43.639667    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:49:49.070104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:49.070039    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"]
Oct 30 19:49:49.070615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:49.070357    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" containerID="cri-o://8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5" gracePeriod=30
Oct 30 19:49:52.999982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:52.999927    6777 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:53.000402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:53.000011    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:54.425622 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.425593    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:49:54.533500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533461    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-envoy\") pod \"e42325e6-a838-4786-9c57-08fa864e811a\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") "
Oct 30 19:49:54.533728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533550    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-kube-api-access-bwr8s\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-kube-api-access-bwr8s\") pod \"e42325e6-a838-4786-9c57-08fa864e811a\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") "
Oct 30 19:49:54.533728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533571    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42325e6-a838-4786-9c57-08fa864e811a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42325e6-a838-4786-9c57-08fa864e811a-istiod-ca-cert\") pod \"e42325e6-a838-4786-9c57-08fa864e811a\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") "
Oct 30 19:49:54.533728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533593    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-socket\") pod \"e42325e6-a838-4786-9c57-08fa864e811a\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") "
Oct 30 19:49:54.533728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533614    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-certs\") pod \"e42325e6-a838-4786-9c57-08fa864e811a\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") "
Oct 30 19:49:54.533728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533639    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-data\") pod \"e42325e6-a838-4786-9c57-08fa864e811a\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") "
Oct 30 19:49:54.533728 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533695    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-credential-socket\") pod \"e42325e6-a838-4786-9c57-08fa864e811a\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") "
Oct 30 19:49:54.534104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533773    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e42325e6-a838-4786-9c57-08fa864e811a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42325e6-a838-4786-9c57-08fa864e811a-istio-podinfo\") pod \"e42325e6-a838-4786-9c57-08fa864e811a\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") "
Oct 30 19:49:54.534104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533801    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-istio-token\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-istio-token\") pod \"e42325e6-a838-4786-9c57-08fa864e811a\" (UID: \"e42325e6-a838-4786-9c57-08fa864e811a\") "
Oct 30 19:49:54.534104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533939    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-socket" pod "e42325e6-a838-4786-9c57-08fa864e811a" (UID: "e42325e6-a838-4786-9c57-08fa864e811a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:54.534104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.533993    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-data" pod "e42325e6-a838-4786-9c57-08fa864e811a" (UID: "e42325e6-a838-4786-9c57-08fa864e811a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:54.534104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.534042    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42325e6-a838-4786-9c57-08fa864e811a-istiod-ca-cert" pod "e42325e6-a838-4786-9c57-08fa864e811a" (UID: "e42325e6-a838-4786-9c57-08fa864e811a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:54.534320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.534218    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-credential-socket" pod "e42325e6-a838-4786-9c57-08fa864e811a" (UID: "e42325e6-a838-4786-9c57-08fa864e811a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:54.534320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.534264    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42325e6-a838-4786-9c57-08fa864e811a-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:54.534320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.534290    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:54.534320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.534319    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:54.534472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.534331    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-certs" pod "e42325e6-a838-4786-9c57-08fa864e811a" (UID: "e42325e6-a838-4786-9c57-08fa864e811a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:54.535911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.535883    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-envoy" pod "e42325e6-a838-4786-9c57-08fa864e811a" (UID: "e42325e6-a838-4786-9c57-08fa864e811a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:54.535911 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.535902    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-kube-api-access-bwr8s" pod "e42325e6-a838-4786-9c57-08fa864e811a" (UID: "e42325e6-a838-4786-9c57-08fa864e811a"). InnerVolumeSpecName "kube-api-access-bwr8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:54.536140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.535977    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-istio-token" pod "e42325e6-a838-4786-9c57-08fa864e811a" (UID: "e42325e6-a838-4786-9c57-08fa864e811a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:54.536260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.536246    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e42325e6-a838-4786-9c57-08fa864e811a-istio-podinfo" pod "e42325e6-a838-4786-9c57-08fa864e811a" (UID: "e42325e6-a838-4786-9c57-08fa864e811a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:54.635434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.635337    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:54.635434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.635373    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42325e6-a838-4786-9c57-08fa864e811a-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:54.635434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.635382    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:54.635434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.635390    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:54.635434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.635399    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwr8s\" (UniqueName: \"kubernetes.io/projected/e42325e6-a838-4786-9c57-08fa864e811a-kube-api-access-bwr8s\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:54.635434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:54.635410    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42325e6-a838-4786-9c57-08fa864e811a-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:55.181411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.181373    6777 generic.go:356] "Generic (PLEG): container finished" podID="e42325e6-a838-4786-9c57-08fa864e811a" containerID="8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5" exitCode=0
Oct 30 19:49:55.181594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.181423    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"
Oct 30 19:49:55.181594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.181450    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" event={"ID":"e42325e6-a838-4786-9c57-08fa864e811a","Type":"ContainerDied","Data":"8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5"}
Oct 30 19:49:55.181594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.181511    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486" event={"ID":"e42325e6-a838-4786-9c57-08fa864e811a","Type":"ContainerDied","Data":"77a334cbf054c42a5bf3d46700953008a34d7eb960196528124a22d0cd9f2867"}
Oct 30 19:49:55.181594 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.181533    6777 scope.go:122] "RemoveContainer" containerID="8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5"
Oct 30 19:49:55.192301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.192098    6777 scope.go:122] "RemoveContainer" containerID="6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562"
Oct 30 19:49:55.200980 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.200960    6777 scope.go:122] "RemoveContainer" containerID="8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5"
Oct 30 19:49:55.201272 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:55.201253    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5\": container with ID starting with 8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5 not found: ID does not exist" containerID="8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5"
Oct 30 19:49:55.201328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.201283    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5"} err="failed to get container status \"8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5\": rpc error: code = NotFound desc = could not find container \"8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5\": container with ID starting with 8f10feee90f02394447e2da6eb0c3ed5bad3370fa5e64ef68f4c9501c5c0ffb5 not found: ID does not exist"
Oct 30 19:49:55.201328 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.201300    6777 scope.go:122] "RemoveContainer" containerID="6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562"
Oct 30 19:49:55.201618 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:49:55.201553    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562\": container with ID starting with 6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562 not found: ID does not exist" containerID="6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562"
Oct 30 19:49:55.201687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.201635    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562"} err="failed to get container status \"6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562\": rpc error: code = NotFound desc = could not find container \"6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562\": container with ID starting with 6d9269d167b93d8ae195bff02383775819ce404a77c7264c2a04b18722325562 not found: ID does not exist"
Oct 30 19:49:55.205024 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.204973    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"]
Oct 30 19:49:55.217461 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:55.217432    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-l4486"]
Oct 30 19:49:56.112812 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:49:56.112780    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42325e6-a838-4786-9c57-08fa864e811a" path="/var/lib/kubelet/pods/e42325e6-a838-4786-9c57-08fa864e811a/volumes"
Oct 30 19:50:06.305078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:06.305029    6777 scope.go:122] "RemoveContainer" containerID="c79d6da5dbae0a246998bff7a80ddf23b00e74a800a19f11339295f7e6bb63ab"
Oct 30 19:50:06.314050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:06.314028    6777 scope.go:122] "RemoveContainer" containerID="fb23294e796cb44f42d9054a9c8527cd6ada56405f1e23bb9e7c680150eca4dc"
Oct 30 19:50:11.593905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.593863    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"]
Oct 30 19:50:11.594340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594323    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy"
Oct 30 19:50:11.594387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594342    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy"
Oct 30 19:50:11.594387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594360    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ba70864a-66bd-44b6-9923-0ac9f00260fd" containerName="payload-processing"
Oct 30 19:50:11.594387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594365    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba70864a-66bd-44b6-9923-0ac9f00260fd" containerName="payload-processing"
Oct 30 19:50:11.594387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594376    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy"
Oct 30 19:50:11.594387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594383    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594391    6777 cpu_manager.go:414] "containerMap: removing container" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594396    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594402    6777 cpu_manager.go:414] "containerMap: removing container" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594408    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594427    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594431    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594463    6777 cpu_manager.go:414] "containerMap: removing container" podUID="d9bbcbd7-69f3-402c-8206-2c050904b75d" containerName="revoke-keys"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594467    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bbcbd7-69f3-402c-8206-2c050904b75d" containerName="revoke-keys"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594538    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a01e0ca-b6ee-44c6-be57-f94a25d39f21" containerName="istio-proxy"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594545    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe94c2e-56f0-4979-8b7c-d83baf87ea6a" containerName="main"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594551    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff91e8e-7ada-4741-815e-06c9955b4151" containerName="main"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594558    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7489c99-67e2-438e-9593-3f552755ff5c" containerName="istio-proxy"
Oct 30 19:50:11.594570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594570    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bbcbd7-69f3-402c-8206-2c050904b75d" containerName="revoke-keys"
Oct 30 19:50:11.595008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594578    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy"
Oct 30 19:50:11.595008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594586    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy"
Oct 30 19:50:11.595008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.594591    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba70864a-66bd-44b6-9923-0ac9f00260fd" containerName="payload-processing"
Oct 30 19:50:11.599776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.599750    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.611942 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.611907    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"]
Oct 30 19:50:11.635813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.635778    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"]
Oct 30 19:50:11.636529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.636498    6777 cpu_manager.go:414] "containerMap: removing container" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy"
Oct 30 19:50:11.636529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.636527    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42325e6-a838-4786-9c57-08fa864e811a" containerName="istio-proxy"
Oct 30 19:50:11.640314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.640289    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.651480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.651448    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"]
Oct 30 19:50:11.710118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710078    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e579180c-22b8-4904-b2a3-acad28497f49-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.710118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710116    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.710407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710139    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.710407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710275    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.710407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710328    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e579180c-22b8-4904-b2a3-acad28497f49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.710407 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710389    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.710546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710424    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.710546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710445    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.710546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710519    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtphg\" (UniqueName: \"kubernetes.io/projected/e579180c-22b8-4904-b2a3-acad28497f49-kube-api-access-dtphg\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.710644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710557    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.710644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710616    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.710711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710679    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.710711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710701    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.710804 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710725    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.710843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710803    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e579180c-22b8-4904-b2a3-acad28497f49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.710889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710867    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.710930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710889    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hs4m\" (UniqueName: \"kubernetes.io/projected/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-kube-api-access-9hs4m\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.710971 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.710952    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.812478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812435    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.812478 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812484    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.812779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812515    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e579180c-22b8-4904-b2a3-acad28497f49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.812779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812583    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.812779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812610    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hs4m\" (UniqueName: \"kubernetes.io/projected/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-kube-api-access-9hs4m\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.812779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812643    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.812779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812695    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e579180c-22b8-4904-b2a3-acad28497f49-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.812779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812719    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.812779 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812745    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812781    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812827    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e579180c-22b8-4904-b2a3-acad28497f49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812876    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812906    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812930    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812945    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812958    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtphg\" (UniqueName: \"kubernetes.io/projected/e579180c-22b8-4904-b2a3-acad28497f49-kube-api-access-dtphg\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.812984    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.813020    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.813070    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.813122    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.813214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.813207    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.813871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.813231    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.813871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.813431    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.813871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.813685    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.813871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.813842    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.814104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.813918    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e579180c-22b8-4904-b2a3-acad28497f49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.814178 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.814108    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.814242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.814219    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.815483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.815458    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e579180c-22b8-4904-b2a3-acad28497f49-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.815643 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.815618    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e579180c-22b8-4904-b2a3-acad28497f49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.815772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.815754    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.815901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.815883    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.825805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.825737    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtphg\" (UniqueName: \"kubernetes.io/projected/e579180c-22b8-4904-b2a3-acad28497f49-kube-api-access-dtphg\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.825970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.825847    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hs4m\" (UniqueName: \"kubernetes.io/projected/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-kube-api-access-9hs4m\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.825970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.825881    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6b2aed-dbe2-4934-8c89-00f2f35ce325-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-m5mbs\" (UID: \"ae6b2aed-dbe2-4934-8c89-00f2f35ce325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.825970 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.825944    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e579180c-22b8-4904-b2a3-acad28497f49-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-xlxqw\" (UID: \"e579180c-22b8-4904-b2a3-acad28497f49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:11.916111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.915980    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:11.955809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:11.955766    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:12.208198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.208168    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"]
Oct 30 19:50:12.210416 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:50:12.210389    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6b2aed_dbe2_4934_8c89_00f2f35ce325.slice/crio-ca60f3975df540244d19442f42309eed58c149cbe518863f1a1321abc408447c WatchSource:0}: Error finding container ca60f3975df540244d19442f42309eed58c149cbe518863f1a1321abc408447c: Status 404 returned error can't find the container with id ca60f3975df540244d19442f42309eed58c149cbe518863f1a1321abc408447c
Oct 30 19:50:12.212708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.212672    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:12.212816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.212740    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:12.212816 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.212769    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:12.263408 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.263369    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"]
Oct 30 19:50:12.265902 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:50:12.265870    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode579180c_22b8_4904_b2a3_acad28497f49.slice/crio-271f067e22674dca996cf5e72e4b4743b1e401312fe7cc099545f9322b2e3ea9 WatchSource:0}: Error finding container 271f067e22674dca996cf5e72e4b4743b1e401312fe7cc099545f9322b2e3ea9: Status 404 returned error can't find the container with id 271f067e22674dca996cf5e72e4b4743b1e401312fe7cc099545f9322b2e3ea9
Oct 30 19:50:12.268329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.268288    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:12.268434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.268365    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:12.268434 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.268397    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:50:12.282583 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.282542    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" event={"ID":"ae6b2aed-dbe2-4934-8c89-00f2f35ce325","Type":"ContainerStarted","Data":"ca60f3975df540244d19442f42309eed58c149cbe518863f1a1321abc408447c"}
Oct 30 19:50:12.284114 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:12.284089    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" event={"ID":"e579180c-22b8-4904-b2a3-acad28497f49","Type":"ContainerStarted","Data":"271f067e22674dca996cf5e72e4b4743b1e401312fe7cc099545f9322b2e3ea9"}
Oct 30 19:50:13.302185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:13.302143    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" event={"ID":"ae6b2aed-dbe2-4934-8c89-00f2f35ce325","Type":"ContainerStarted","Data":"0ac827d50b1a89aeb919be2a0d52f501d9122e3695a4b24a6218325c6102e162"}
Oct 30 19:50:13.304822 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:13.304792    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" event={"ID":"e579180c-22b8-4904-b2a3-acad28497f49","Type":"ContainerStarted","Data":"fd092f0d7d5b7b3d65dadcf71df725ae07d40ce5e9aafe2b7e25a591ae2bf2c1"}
Oct 30 19:50:13.329654 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:13.329591    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podStartSLOduration=2.329572781 podStartE2EDuration="2.329572781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:13.328154877 +0000 UTC m=+2595.325070265" watchObservedRunningTime="2026-10-30 19:50:13.329572781 +0000 UTC m=+2595.326488166"
Oct 30 19:50:13.639213 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:50:13.639135    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:50:13.916452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:13.916358    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:13.917890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:13.917871    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-m5mbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:50:13.917947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:13.917915    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podUID="ae6b2aed-dbe2-4934-8c89-00f2f35ce325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Oct 30 19:50:13.981758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:13.981723    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:13.994319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:13.994278    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xlxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 30 19:50:13.994978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:13.994942    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podUID="e579180c-22b8-4904-b2a3-acad28497f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 30 19:50:14.333179 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.333117    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podStartSLOduration=3.333097484 podStartE2EDuration="3.333097484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:13.371330372 +0000 UTC m=+2595.368245757" watchObservedRunningTime="2026-10-30 19:50:14.333097484 +0000 UTC m=+2596.330012871"
Oct 30 19:50:14.333703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.333414    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"]
Oct 30 19:50:14.336866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.336840    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.344563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.344529    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:50:14.344563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.344529    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:50:14.346389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.346251    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:50:14.346389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.346314    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:50:14.346973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.346418    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"]
Oct 30 19:50:14.440198 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.440154    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lgtkg\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.440365 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.440241    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lgtkg\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.440585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.440564    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx66x\" (UniqueName: \"kubernetes.io/projected/179707e6-345c-4b9d-a10b-2fa007ae5bb0-kube-api-access-jx66x\") pod \"maas-api-revoke-sub-85237e45-lgtkg\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.541898 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.541850    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx66x\" (UniqueName: \"kubernetes.io/projected/179707e6-345c-4b9d-a10b-2fa007ae5bb0-kube-api-access-jx66x\") pod \"maas-api-revoke-sub-85237e45-lgtkg\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.542129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.541950    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lgtkg\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.542129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.542011    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lgtkg\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.542675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.542639    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lgtkg\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.545614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.545583    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lgtkg\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.592134 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.591576    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx66x\" (UniqueName: \"kubernetes.io/projected/179707e6-345c-4b9d-a10b-2fa007ae5bb0-kube-api-access-jx66x\") pod \"maas-api-revoke-sub-85237e45-lgtkg\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.709586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.709535    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:14.916939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.916845    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-m5mbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:50:14.916939 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.916908    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podUID="ae6b2aed-dbe2-4934-8c89-00f2f35ce325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Oct 30 19:50:14.985296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.984413    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xlxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 30 19:50:14.985296 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:14.984471    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podUID="e579180c-22b8-4904-b2a3-acad28497f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 30 19:50:15.042128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:15.042101    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"]
Oct 30 19:50:15.043916 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:50:15.043321    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179707e6_345c_4b9d_a10b_2fa007ae5bb0.slice/crio-ee10aa2b42c372c7222ba41c315703a094f70ccaa867dec27f20a2bc18c2edff WatchSource:0}: Error finding container ee10aa2b42c372c7222ba41c315703a094f70ccaa867dec27f20a2bc18c2edff: Status 404 returned error can't find the container with id ee10aa2b42c372c7222ba41c315703a094f70ccaa867dec27f20a2bc18c2edff
Oct 30 19:50:15.317310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:15.317272    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg" event={"ID":"179707e6-345c-4b9d-a10b-2fa007ae5bb0","Type":"ContainerStarted","Data":"ee10aa2b42c372c7222ba41c315703a094f70ccaa867dec27f20a2bc18c2edff"}
Oct 30 19:50:15.916765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:15.916724    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-m5mbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:50:15.917283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:15.916784    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podUID="ae6b2aed-dbe2-4934-8c89-00f2f35ce325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Oct 30 19:50:15.976084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:15.970582    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xlxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 30 19:50:15.976084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:15.970654    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podUID="e579180c-22b8-4904-b2a3-acad28497f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 30 19:50:16.324226 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:16.324186    6777 generic.go:356] "Generic (PLEG): container finished" podID="179707e6-345c-4b9d-a10b-2fa007ae5bb0" containerID="d9606f14f6021ea060aa3deca1eee78f26ddfdf6c129c2c8d83c45845b621266" exitCode=0
Oct 30 19:50:16.324387 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:16.324262    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg" event={"ID":"179707e6-345c-4b9d-a10b-2fa007ae5bb0","Type":"ContainerDied","Data":"d9606f14f6021ea060aa3deca1eee78f26ddfdf6c129c2c8d83c45845b621266"}
Oct 30 19:50:16.916788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:16.916747    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-m5mbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:50:16.917413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:16.917379    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podUID="ae6b2aed-dbe2-4934-8c89-00f2f35ce325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Oct 30 19:50:16.968083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:16.958029    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xlxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 30 19:50:16.968083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:16.958105    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podUID="e579180c-22b8-4904-b2a3-acad28497f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 30 19:50:17.578656 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.578284    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:17.717325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.717220    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-openshift-ca\") pod \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") "
Oct 30 19:50:17.717556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.717533    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179707e6-345c-4b9d-a10b-2fa007ae5bb0-kube-api-access-jx66x\" (UniqueName: \"kubernetes.io/projected/179707e6-345c-4b9d-a10b-2fa007ae5bb0-kube-api-access-jx66x\") pod \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") "
Oct 30 19:50:17.717621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.717606    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-xks-ca\") pod \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\" (UID: \"179707e6-345c-4b9d-a10b-2fa007ae5bb0\") "
Oct 30 19:50:17.717823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.717805    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-openshift-ca" pod "179707e6-345c-4b9d-a10b-2fa007ae5bb0" (UID: "179707e6-345c-4b9d-a10b-2fa007ae5bb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:17.717935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.717922    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:17.721086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.721040    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-xks-ca" pod "179707e6-345c-4b9d-a10b-2fa007ae5bb0" (UID: "179707e6-345c-4b9d-a10b-2fa007ae5bb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:50:17.721086 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.721074    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179707e6-345c-4b9d-a10b-2fa007ae5bb0-kube-api-access-jx66x" pod "179707e6-345c-4b9d-a10b-2fa007ae5bb0" (UID: "179707e6-345c-4b9d-a10b-2fa007ae5bb0"). InnerVolumeSpecName "kube-api-access-jx66x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:17.818854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.818811    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179707e6-345c-4b9d-a10b-2fa007ae5bb0-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:17.818854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.818847    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx66x\" (UniqueName: \"kubernetes.io/projected/179707e6-345c-4b9d-a10b-2fa007ae5bb0-kube-api-access-jx66x\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:17.917501 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.917450    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-m5mbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:50:17.918083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.917513    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podUID="ae6b2aed-dbe2-4934-8c89-00f2f35ce325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Oct 30 19:50:17.959091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.957227    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xlxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 30 19:50:17.959091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:17.957292    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podUID="e579180c-22b8-4904-b2a3-acad28497f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 30 19:50:18.337308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:18.337271    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"
Oct 30 19:50:18.337308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:18.337297    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg" event={"ID":"179707e6-345c-4b9d-a10b-2fa007ae5bb0","Type":"ContainerDied","Data":"ee10aa2b42c372c7222ba41c315703a094f70ccaa867dec27f20a2bc18c2edff"}
Oct 30 19:50:18.337584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:18.337351    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee10aa2b42c372c7222ba41c315703a094f70ccaa867dec27f20a2bc18c2edff"
Oct 30 19:50:18.917084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:18.917031    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-m5mbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:50:18.917325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:18.917118    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podUID="ae6b2aed-dbe2-4934-8c89-00f2f35ce325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Oct 30 19:50:18.957256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:18.957213    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xlxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body=
Oct 30 19:50:18.957669 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:18.957286    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podUID="e579180c-22b8-4904-b2a3-acad28497f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused"
Oct 30 19:50:20.918369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:20.918321    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-m5mbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:20.919044 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:20.918394    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podUID="ae6b2aed-dbe2-4934-8c89-00f2f35ce325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:20.957113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:20.957067    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xlxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:20.957334 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:20.957143    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podUID="e579180c-22b8-4904-b2a3-acad28497f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:21.962216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:21.962166    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-m5mbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:21.962777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:21.962240    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podUID="ae6b2aed-dbe2-4934-8c89-00f2f35ce325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:21.998301 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:21.998258    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xlxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:21.998662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:21.998610    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podUID="e579180c-22b8-4904-b2a3-acad28497f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:22.990184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:22.990142    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-m5mbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:22.990663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:22.990203    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs" podUID="ae6b2aed-dbe2-4934-8c89-00f2f35ce325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:23.004839 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:23.004802    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xlxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:23.005019 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:23.004864    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw" podUID="e579180c-22b8-4904-b2a3-acad28497f49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:23.009335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:23.009305    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:23.009735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:23.009708    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:23.010563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:23.010538    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:23.010670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:23.010618    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xlxqw"
Oct 30 19:50:23.010981 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:23.010964    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:23.011855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:23.011836    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-m5mbs"
Oct 30 19:50:43.640042 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:50:43.639994    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:50:53.769513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:53.769477    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:50:58.420890 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.420856    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"]
Oct 30 19:50:58.421326 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.421312    6777 cpu_manager.go:414] "containerMap: removing container" podUID="179707e6-345c-4b9d-a10b-2fa007ae5bb0" containerName="revoke-keys"
Oct 30 19:50:58.421372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.421326    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179707e6-345c-4b9d-a10b-2fa007ae5bb0" containerName="revoke-keys"
Oct 30 19:50:58.421415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.421397    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="179707e6-345c-4b9d-a10b-2fa007ae5bb0" containerName="revoke-keys"
Oct 30 19:50:58.424409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.424391    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.429466 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.429440    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:50:58.430888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.430861    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:50:58.430888 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.430885    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:50:58.431113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.431001    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:50:58.438356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.438325    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"]
Oct 30 19:50:58.466350 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.466317    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-457fg\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.466537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.466370    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-457fg\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.466537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.466464    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfnh6\" (UniqueName: \"kubernetes.io/projected/ed0c4eed-9b82-4e95-af23-085885d63338-kube-api-access-zfnh6\") pod \"maas-api-revoke-sub-c1e271a9-457fg\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.567221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.567179    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-457fg\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.567427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.567229    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-457fg\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.567427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.567254    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfnh6\" (UniqueName: \"kubernetes.io/projected/ed0c4eed-9b82-4e95-af23-085885d63338-kube-api-access-zfnh6\") pod \"maas-api-revoke-sub-c1e271a9-457fg\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.567995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.567967    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-457fg\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.569618 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.569594    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-457fg\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.579947 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.579918    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfnh6\" (UniqueName: \"kubernetes.io/projected/ed0c4eed-9b82-4e95-af23-085885d63338-kube-api-access-zfnh6\") pod \"maas-api-revoke-sub-c1e271a9-457fg\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:58.739872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:58.739784    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:50:59.016170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:59.016139    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"]
Oct 30 19:50:59.017910 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:50:59.017880    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0c4eed_9b82_4e95_af23_085885d63338.slice/crio-79179ecccfa28e2cb4203752788033c5b6f9b30f67153bd0131a0ce6e706d29b WatchSource:0}: Error finding container 79179ecccfa28e2cb4203752788033c5b6f9b30f67153bd0131a0ce6e706d29b: Status 404 returned error can't find the container with id 79179ecccfa28e2cb4203752788033c5b6f9b30f67153bd0131a0ce6e706d29b
Oct 30 19:50:59.666144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:59.666107    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg" event={"ID":"ed0c4eed-9b82-4e95-af23-085885d63338","Type":"ContainerStarted","Data":"c1a4bf6ea589d219bb4e7831aea284ac8a64aa7bc7697fd73973c539fa90a0e1"}
Oct 30 19:50:59.666575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:59.666167    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg" event={"ID":"ed0c4eed-9b82-4e95-af23-085885d63338","Type":"ContainerStarted","Data":"79179ecccfa28e2cb4203752788033c5b6f9b30f67153bd0131a0ce6e706d29b"}
Oct 30 19:50:59.687216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:50:59.687165    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg" podStartSLOduration=1.6871498919999999 podStartE2EDuration="1.687149892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:59.685948798 +0000 UTC m=+2641.682864181" watchObservedRunningTime="2026-10-30 19:50:59.687149892 +0000 UTC m=+2641.684065275"
Oct 30 19:51:00.673162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:00.673127    6777 generic.go:356] "Generic (PLEG): container finished" podID="ed0c4eed-9b82-4e95-af23-085885d63338" containerID="c1a4bf6ea589d219bb4e7831aea284ac8a64aa7bc7697fd73973c539fa90a0e1" exitCode=0
Oct 30 19:51:00.673603 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:00.673214    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg" event={"ID":"ed0c4eed-9b82-4e95-af23-085885d63338","Type":"ContainerDied","Data":"c1a4bf6ea589d219bb4e7831aea284ac8a64aa7bc7697fd73973c539fa90a0e1"}
Oct 30 19:51:01.820695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:01.820665    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:51:01.905580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:01.905535    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-xks-ca\") pod \"ed0c4eed-9b82-4e95-af23-085885d63338\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") "
Oct 30 19:51:01.905774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:01.905639    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0c4eed-9b82-4e95-af23-085885d63338-kube-api-access-zfnh6\" (UniqueName: \"kubernetes.io/projected/ed0c4eed-9b82-4e95-af23-085885d63338-kube-api-access-zfnh6\") pod \"ed0c4eed-9b82-4e95-af23-085885d63338\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") "
Oct 30 19:51:01.905774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:01.905748    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-openshift-ca\") pod \"ed0c4eed-9b82-4e95-af23-085885d63338\" (UID: \"ed0c4eed-9b82-4e95-af23-085885d63338\") "
Oct 30 19:51:01.906154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:01.906122    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-openshift-ca" pod "ed0c4eed-9b82-4e95-af23-085885d63338" (UID: "ed0c4eed-9b82-4e95-af23-085885d63338"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:01.907670 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:01.907651    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-xks-ca" pod "ed0c4eed-9b82-4e95-af23-085885d63338" (UID: "ed0c4eed-9b82-4e95-af23-085885d63338"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:01.907807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:01.907790    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0c4eed-9b82-4e95-af23-085885d63338-kube-api-access-zfnh6" pod "ed0c4eed-9b82-4e95-af23-085885d63338" (UID: "ed0c4eed-9b82-4e95-af23-085885d63338"). InnerVolumeSpecName "kube-api-access-zfnh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:02.007589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:02.007497    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfnh6\" (UniqueName: \"kubernetes.io/projected/ed0c4eed-9b82-4e95-af23-085885d63338-kube-api-access-zfnh6\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:02.007589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:02.007535    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:02.007589 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:02.007548    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0c4eed-9b82-4e95-af23-085885d63338-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:02.685608 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:02.685574    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"
Oct 30 19:51:02.685805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:02.685609    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg" event={"ID":"ed0c4eed-9b82-4e95-af23-085885d63338","Type":"ContainerDied","Data":"79179ecccfa28e2cb4203752788033c5b6f9b30f67153bd0131a0ce6e706d29b"}
Oct 30 19:51:02.685805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:02.685683    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79179ecccfa28e2cb4203752788033c5b6f9b30f67153bd0131a0ce6e706d29b"
Oct 30 19:51:06.396256 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:06.396222    6777 scope.go:122] "RemoveContainer" containerID="c21652150aabb9214115cb43d1a48e0bb819a1e0d34ca3155bb14c185e61047e"
Oct 30 19:51:09.136252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:09.136212    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:51:12.473118 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:12.470032    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:51:13.733591 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:51:13.733521    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:51:17.098149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.098114    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"]
Oct 30 19:51:17.098605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.098589    6777 cpu_manager.go:414] "containerMap: removing container" podUID="ed0c4eed-9b82-4e95-af23-085885d63338" containerName="revoke-keys"
Oct 30 19:51:17.098652 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.098608    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0c4eed-9b82-4e95-af23-085885d63338" containerName="revoke-keys"
Oct 30 19:51:17.098693 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.098674    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0c4eed-9b82-4e95-af23-085885d63338" containerName="revoke-keys"
Oct 30 19:51:17.101955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.101936    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.109577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.109546    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:17.110409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.110289    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:51:17.110409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.110375    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:51:17.110409 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.110394    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:51:17.113144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.113111    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"]
Oct 30 19:51:17.218358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.218309    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6d7\" (UniqueName: \"kubernetes.io/projected/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-kube-api-access-zn6d7\") pod \"maas-api-revoke-sub-2e1f5b5f-jv7h9\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.218553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.218371    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jv7h9\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.218553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.218531    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jv7h9\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.319626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.319583    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6d7\" (UniqueName: \"kubernetes.io/projected/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-kube-api-access-zn6d7\") pod \"maas-api-revoke-sub-2e1f5b5f-jv7h9\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.319626 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.319634    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jv7h9\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.319870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.319706    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jv7h9\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.320347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.320326    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jv7h9\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.322238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.322216    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jv7h9\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.332711 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.332688    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6d7\" (UniqueName: \"kubernetes.io/projected/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-kube-api-access-zn6d7\") pod \"maas-api-revoke-sub-2e1f5b5f-jv7h9\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.418445 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.418346    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:17.693143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.693116    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"]
Oct 30 19:51:17.695374 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:51:17.695344    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b47a90c_88f3_4c12_873c_f5dd4cc5ab63.slice/crio-29b3eec81730037737cfb77135248a0907e69ecd0023b10c4964c8375da2c459 WatchSource:0}: Error finding container 29b3eec81730037737cfb77135248a0907e69ecd0023b10c4964c8375da2c459: Status 404 returned error can't find the container with id 29b3eec81730037737cfb77135248a0907e69ecd0023b10c4964c8375da2c459
Oct 30 19:51:17.800813 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:17.800776    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9" event={"ID":"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63","Type":"ContainerStarted","Data":"29b3eec81730037737cfb77135248a0907e69ecd0023b10c4964c8375da2c459"}
Oct 30 19:51:18.807910 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:18.807815    6777 generic.go:356] "Generic (PLEG): container finished" podID="2b47a90c-88f3-4c12-873c-f5dd4cc5ab63" containerID="f02aa6443a7d6482ad0f026eb0bed723ab551a3236f804e65bc3fe90736f3228" exitCode=0
Oct 30 19:51:18.807910 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:18.807902    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9" event={"ID":"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63","Type":"ContainerDied","Data":"f02aa6443a7d6482ad0f026eb0bed723ab551a3236f804e65bc3fe90736f3228"}
Oct 30 19:51:19.947553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:19.947528    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:20.047152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.047114    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-xks-ca\") pod \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") "
Oct 30 19:51:20.047337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.047229    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-openshift-ca\") pod \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") "
Oct 30 19:51:20.047337 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.047302    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-kube-api-access-zn6d7\" (UniqueName: \"kubernetes.io/projected/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-kube-api-access-zn6d7\") pod \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\" (UID: \"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63\") "
Oct 30 19:51:20.047640 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.047616    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-openshift-ca" pod "2b47a90c-88f3-4c12-873c-f5dd4cc5ab63" (UID: "2b47a90c-88f3-4c12-873c-f5dd4cc5ab63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:20.049293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.049266    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-xks-ca" pod "2b47a90c-88f3-4c12-873c-f5dd4cc5ab63" (UID: "2b47a90c-88f3-4c12-873c-f5dd4cc5ab63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:20.049426 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.049306    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-kube-api-access-zn6d7" pod "2b47a90c-88f3-4c12-873c-f5dd4cc5ab63" (UID: "2b47a90c-88f3-4c12-873c-f5dd4cc5ab63"). InnerVolumeSpecName "kube-api-access-zn6d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:20.148724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.148676    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:20.148724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.148717    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn6d7\" (UniqueName: \"kubernetes.io/projected/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-kube-api-access-zn6d7\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:20.148724 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.148731    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:20.820245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.820207    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9" event={"ID":"2b47a90c-88f3-4c12-873c-f5dd4cc5ab63","Type":"ContainerDied","Data":"29b3eec81730037737cfb77135248a0907e69ecd0023b10c4964c8375da2c459"}
Oct 30 19:51:20.820245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.820232    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"
Oct 30 19:51:20.820473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:20.820260    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b3eec81730037737cfb77135248a0907e69ecd0023b10c4964c8375da2c459"
Oct 30 19:51:27.850857 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:27.850811    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:51:28.009870 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.009829    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"]
Oct 30 19:51:28.010348 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.010331    6777 cpu_manager.go:414] "containerMap: removing container" podUID="2b47a90c-88f3-4c12-873c-f5dd4cc5ab63" containerName="revoke-keys"
Oct 30 19:51:28.010419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.010349    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b47a90c-88f3-4c12-873c-f5dd4cc5ab63" containerName="revoke-keys"
Oct 30 19:51:28.010419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.010411    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b47a90c-88f3-4c12-873c-f5dd4cc5ab63" containerName="revoke-keys"
Oct 30 19:51:28.013577 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.013557    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.020596 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.020561    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:51:28.023532 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.023505    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"]
Oct 30 19:51:28.023682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.023653    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:51:28.023682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.023676    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:28.024163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.024141    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:51:28.032537 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.032500    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mjgzk\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.032893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.032872    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mjgzk\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.033140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.033112    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzxc\" (UniqueName: \"kubernetes.io/projected/64e4517f-5386-4fb9-8c67-4026e4fe81b5-kube-api-access-rrzxc\") pod \"maas-api-revoke-sub-ec843018-mjgzk\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.134402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.134296    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mjgzk\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.134402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.134349    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzxc\" (UniqueName: \"kubernetes.io/projected/64e4517f-5386-4fb9-8c67-4026e4fe81b5-kube-api-access-rrzxc\") pod \"maas-api-revoke-sub-ec843018-mjgzk\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.134629 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.134432    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mjgzk\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.135089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.135039    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mjgzk\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.136849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.136826    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mjgzk\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.147312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.147287    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzxc\" (UniqueName: \"kubernetes.io/projected/64e4517f-5386-4fb9-8c67-4026e4fe81b5-kube-api-access-rrzxc\") pod \"maas-api-revoke-sub-ec843018-mjgzk\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.332855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.332820    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:28.610505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.610477    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"]
Oct 30 19:51:28.612950 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:51:28.612923    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e4517f_5386_4fb9_8c67_4026e4fe81b5.slice/crio-7c8228571252daf21627df1acbd4bd6a7985963ec5dd92f6daa2311fda58e1c2 WatchSource:0}: Error finding container 7c8228571252daf21627df1acbd4bd6a7985963ec5dd92f6daa2311fda58e1c2: Status 404 returned error can't find the container with id 7c8228571252daf21627df1acbd4bd6a7985963ec5dd92f6daa2311fda58e1c2
Oct 30 19:51:28.867929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.867892    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk" event={"ID":"64e4517f-5386-4fb9-8c67-4026e4fe81b5","Type":"ContainerStarted","Data":"f5f851bca7360893d053e5bb932bb4c78d8852d161608a288aa32b06d4fe86cd"}
Oct 30 19:51:28.868379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.867949    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk" event={"ID":"64e4517f-5386-4fb9-8c67-4026e4fe81b5","Type":"ContainerStarted","Data":"7c8228571252daf21627df1acbd4bd6a7985963ec5dd92f6daa2311fda58e1c2"}
Oct 30 19:51:28.887647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:28.887592    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk" podStartSLOduration=1.88757976 podStartE2EDuration="1.88757976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:28.886996009 +0000 UTC m=+2670.883911393" watchObservedRunningTime="2026-10-30 19:51:28.88757976 +0000 UTC m=+2670.884495143"
Oct 30 19:51:29.878795 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:29.878752    6777 generic.go:356] "Generic (PLEG): container finished" podID="64e4517f-5386-4fb9-8c67-4026e4fe81b5" containerID="f5f851bca7360893d053e5bb932bb4c78d8852d161608a288aa32b06d4fe86cd" exitCode=0
Oct 30 19:51:29.879421 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:29.878859    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk" event={"ID":"64e4517f-5386-4fb9-8c67-4026e4fe81b5","Type":"ContainerDied","Data":"f5f851bca7360893d053e5bb932bb4c78d8852d161608a288aa32b06d4fe86cd"}
Oct 30 19:51:31.581313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.581278    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:31.829084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.814259    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-openshift-ca\") pod \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") "
Oct 30 19:51:31.830705 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.830676    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e4517f-5386-4fb9-8c67-4026e4fe81b5-kube-api-access-rrzxc\" (UniqueName: \"kubernetes.io/projected/64e4517f-5386-4fb9-8c67-4026e4fe81b5-kube-api-access-rrzxc\") pod \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") "
Oct 30 19:51:31.831429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.831407    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-xks-ca\") pod \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\" (UID: \"64e4517f-5386-4fb9-8c67-4026e4fe81b5\") "
Oct 30 19:51:31.832284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.830006    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-openshift-ca" pod "64e4517f-5386-4fb9-8c67-4026e4fe81b5" (UID: "64e4517f-5386-4fb9-8c67-4026e4fe81b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:31.893099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.884512    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e4517f-5386-4fb9-8c67-4026e4fe81b5-kube-api-access-rrzxc" pod "64e4517f-5386-4fb9-8c67-4026e4fe81b5" (UID: "64e4517f-5386-4fb9-8c67-4026e4fe81b5"). InnerVolumeSpecName "kube-api-access-rrzxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:31.953412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.951015    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrzxc\" (UniqueName: \"kubernetes.io/projected/64e4517f-5386-4fb9-8c67-4026e4fe81b5-kube-api-access-rrzxc\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.953412 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.951077    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:31.970553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.970512    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk" event={"ID":"64e4517f-5386-4fb9-8c67-4026e4fe81b5","Type":"ContainerDied","Data":"7c8228571252daf21627df1acbd4bd6a7985963ec5dd92f6daa2311fda58e1c2"}
Oct 30 19:51:31.970838 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.970819    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c8228571252daf21627df1acbd4bd6a7985963ec5dd92f6daa2311fda58e1c2"
Oct 30 19:51:31.971042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.971024    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"
Oct 30 19:51:31.976283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:31.976228    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-xks-ca" pod "64e4517f-5386-4fb9-8c67-4026e4fe81b5" (UID: "64e4517f-5386-4fb9-8c67-4026e4fe81b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:32.055845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:32.055789    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e4517f-5386-4fb9-8c67-4026e4fe81b5-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:32.430084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:32.429209    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:32.430084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:32.429311    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:40.375471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:40.375421    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:51:43.642525 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:51:43.642464    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:51:54.424389 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.424343    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"]
Oct 30 19:51:54.425010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.424987    6777 cpu_manager.go:414] "containerMap: removing container" podUID="64e4517f-5386-4fb9-8c67-4026e4fe81b5" containerName="revoke-keys"
Oct 30 19:51:54.425158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.425013    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e4517f-5386-4fb9-8c67-4026e4fe81b5" containerName="revoke-keys"
Oct 30 19:51:54.425158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.425132    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e4517f-5386-4fb9-8c67-4026e4fe81b5" containerName="revoke-keys"
Oct 30 19:51:54.428877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.428853    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.435493 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.435466    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:51:54.435680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.435529    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:54.437246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.437159    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:51:54.437355 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.437266    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:51:54.437433 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.437410    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"]
Oct 30 19:51:54.562885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.562844    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrk5s\" (UniqueName: \"kubernetes.io/projected/c227135d-ab77-463d-936e-a574d9c3ac2c-kube-api-access-vrk5s\") pod \"maas-api-revoke-sub-046e1343-5hw26\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.563100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.562918    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5hw26\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.563100 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.563052    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5hw26\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.664352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.664310    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5hw26\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.664352 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.664357    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5hw26\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.664584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.664456    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrk5s\" (UniqueName: \"kubernetes.io/projected/c227135d-ab77-463d-936e-a574d9c3ac2c-kube-api-access-vrk5s\") pod \"maas-api-revoke-sub-046e1343-5hw26\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.665133 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.665106    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5hw26\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.666826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.666804    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5hw26\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.677288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.677223    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrk5s\" (UniqueName: \"kubernetes.io/projected/c227135d-ab77-463d-936e-a574d9c3ac2c-kube-api-access-vrk5s\") pod \"maas-api-revoke-sub-046e1343-5hw26\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:54.744282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:54.744251    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:55.028037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:55.028010    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"]
Oct 30 19:51:55.030467 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:51:55.030440    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc227135d_ab77_463d_936e_a574d9c3ac2c.slice/crio-631d707c53afd30bf12dad9cdba1a2448eda615e3fd7406b116f2cb073bf4ca3 WatchSource:0}: Error finding container 631d707c53afd30bf12dad9cdba1a2448eda615e3fd7406b116f2cb073bf4ca3: Status 404 returned error can't find the container with id 631d707c53afd30bf12dad9cdba1a2448eda615e3fd7406b116f2cb073bf4ca3
Oct 30 19:51:55.136678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:55.136643    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26" event={"ID":"c227135d-ab77-463d-936e-a574d9c3ac2c","Type":"ContainerStarted","Data":"631d707c53afd30bf12dad9cdba1a2448eda615e3fd7406b116f2cb073bf4ca3"}
Oct 30 19:51:56.143707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:56.143671    6777 generic.go:356] "Generic (PLEG): container finished" podID="c227135d-ab77-463d-936e-a574d9c3ac2c" containerID="83fbb17dfed1819320b370c4528d9f080a5ff3fdfc8ffc2350e3246264f1fe19" exitCode=0
Oct 30 19:51:56.144131 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:56.143757    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26" event={"ID":"c227135d-ab77-463d-936e-a574d9c3ac2c","Type":"ContainerDied","Data":"83fbb17dfed1819320b370c4528d9f080a5ff3fdfc8ffc2350e3246264f1fe19"}
Oct 30 19:51:57.294901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.294872    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:51:57.392745 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.392710    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-openshift-ca\") pod \"c227135d-ab77-463d-936e-a574d9c3ac2c\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") "
Oct 30 19:51:57.392745 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.392750    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c227135d-ab77-463d-936e-a574d9c3ac2c-kube-api-access-vrk5s\" (UniqueName: \"kubernetes.io/projected/c227135d-ab77-463d-936e-a574d9c3ac2c-kube-api-access-vrk5s\") pod \"c227135d-ab77-463d-936e-a574d9c3ac2c\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") "
Oct 30 19:51:57.393039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.392808    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-xks-ca\") pod \"c227135d-ab77-463d-936e-a574d9c3ac2c\" (UID: \"c227135d-ab77-463d-936e-a574d9c3ac2c\") "
Oct 30 19:51:57.393189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.393159    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-openshift-ca" pod "c227135d-ab77-463d-936e-a574d9c3ac2c" (UID: "c227135d-ab77-463d-936e-a574d9c3ac2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:57.394944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.394917    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-xks-ca" pod "c227135d-ab77-463d-936e-a574d9c3ac2c" (UID: "c227135d-ab77-463d-936e-a574d9c3ac2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:57.395099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.395081    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c227135d-ab77-463d-936e-a574d9c3ac2c-kube-api-access-vrk5s" pod "c227135d-ab77-463d-936e-a574d9c3ac2c" (UID: "c227135d-ab77-463d-936e-a574d9c3ac2c"). InnerVolumeSpecName "kube-api-access-vrk5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:57.493575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.493537    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:57.493575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.493567    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrk5s\" (UniqueName: \"kubernetes.io/projected/c227135d-ab77-463d-936e-a574d9c3ac2c-kube-api-access-vrk5s\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:57.493575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:57.493578    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c227135d-ab77-463d-936e-a574d9c3ac2c-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:58.155990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:58.155951    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26" event={"ID":"c227135d-ab77-463d-936e-a574d9c3ac2c","Type":"ContainerDied","Data":"631d707c53afd30bf12dad9cdba1a2448eda615e3fd7406b116f2cb073bf4ca3"}
Oct 30 19:51:58.156204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:58.156010    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="631d707c53afd30bf12dad9cdba1a2448eda615e3fd7406b116f2cb073bf4ca3"
Oct 30 19:51:58.156204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:51:58.155976    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"
Oct 30 19:52:04.633809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.633768    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"]
Oct 30 19:52:04.634325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.634209    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c227135d-ab77-463d-936e-a574d9c3ac2c" containerName="revoke-keys"
Oct 30 19:52:04.634325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.634222    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c227135d-ab77-463d-936e-a574d9c3ac2c" containerName="revoke-keys"
Oct 30 19:52:04.634325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.634316    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c227135d-ab77-463d-936e-a574d9c3ac2c" containerName="revoke-keys"
Oct 30 19:52:04.637829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.637808    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.645790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.645762    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:52:04.645938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.645846    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:04.646548 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.646522    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"]
Oct 30 19:52:04.676370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.676337    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:52:04.676565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.676387    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:52:04.772579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.772537    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dj8\" (UniqueName: \"kubernetes.io/projected/eccf5bbe-61df-4855-b55c-54af345ef794-kube-api-access-d8dj8\") pod \"maas-api-revoke-sub-d3fba056-t7z6w\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.772783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.772588    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-t7z6w\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.772783 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.772717    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-t7z6w\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.874101 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.874045    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dj8\" (UniqueName: \"kubernetes.io/projected/eccf5bbe-61df-4855-b55c-54af345ef794-kube-api-access-d8dj8\") pod \"maas-api-revoke-sub-d3fba056-t7z6w\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.874335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.874112    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-t7z6w\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.874335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.874174    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-t7z6w\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.874731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.874708    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-t7z6w\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.876458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.876428    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-t7z6w\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.887452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.887388    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dj8\" (UniqueName: \"kubernetes.io/projected/eccf5bbe-61df-4855-b55c-54af345ef794-kube-api-access-d8dj8\") pod \"maas-api-revoke-sub-d3fba056-t7z6w\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:04.952694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:04.952653    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:05.184990 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:05.184890    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:52:05.228694 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:05.228663    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"]
Oct 30 19:52:05.230859 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:52:05.230830    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccf5bbe_61df_4855_b55c_54af345ef794.slice/crio-1a298b379cbd11e33b973d3ecf075d55b71b14feeedaa6198b94cf5c27f4ae63 WatchSource:0}: Error finding container 1a298b379cbd11e33b973d3ecf075d55b71b14feeedaa6198b94cf5c27f4ae63: Status 404 returned error can't find the container with id 1a298b379cbd11e33b973d3ecf075d55b71b14feeedaa6198b94cf5c27f4ae63
Oct 30 19:52:06.206331 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:06.206281    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w" event={"ID":"eccf5bbe-61df-4855-b55c-54af345ef794","Type":"ContainerStarted","Data":"6cc4d5cc7e5dad2bcded37acbaa4880ed56bae7197615704cf1457318f89717a"}
Oct 30 19:52:06.206930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:06.206358    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w" event={"ID":"eccf5bbe-61df-4855-b55c-54af345ef794","Type":"ContainerStarted","Data":"1a298b379cbd11e33b973d3ecf075d55b71b14feeedaa6198b94cf5c27f4ae63"}
Oct 30 19:52:06.231314 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:06.231231    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w" podStartSLOduration=2.231195558 podStartE2EDuration="2.231195558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:06.228273993 +0000 UTC m=+2708.225189378" watchObservedRunningTime="2026-10-30 19:52:06.231195558 +0000 UTC m=+2708.228110944"
Oct 30 19:52:06.419965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:06.419896    6777 scope.go:122] "RemoveContainer" containerID="9fa629c139054aa591fa11dcdf878f358253cf52483fa9133d19dd9105eb2738"
Oct 30 19:52:07.235766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:07.235251    6777 generic.go:356] "Generic (PLEG): container finished" podID="eccf5bbe-61df-4855-b55c-54af345ef794" containerID="6cc4d5cc7e5dad2bcded37acbaa4880ed56bae7197615704cf1457318f89717a" exitCode=0
Oct 30 19:52:07.235766 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:07.235377    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w" event={"ID":"eccf5bbe-61df-4855-b55c-54af345ef794","Type":"ContainerDied","Data":"6cc4d5cc7e5dad2bcded37acbaa4880ed56bae7197615704cf1457318f89717a"}
Oct 30 19:52:08.407214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.407185    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:08.467680 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.467642    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eccf5bbe-61df-4855-b55c-54af345ef794-kube-api-access-d8dj8\" (UniqueName: \"kubernetes.io/projected/eccf5bbe-61df-4855-b55c-54af345ef794-kube-api-access-d8dj8\") pod \"eccf5bbe-61df-4855-b55c-54af345ef794\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") "
Oct 30 19:52:08.467871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.467697    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-xks-ca\") pod \"eccf5bbe-61df-4855-b55c-54af345ef794\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") "
Oct 30 19:52:08.467871 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.467729    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-openshift-ca\") pod \"eccf5bbe-61df-4855-b55c-54af345ef794\" (UID: \"eccf5bbe-61df-4855-b55c-54af345ef794\") "
Oct 30 19:52:08.468167 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.468143    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-openshift-ca" pod "eccf5bbe-61df-4855-b55c-54af345ef794" (UID: "eccf5bbe-61df-4855-b55c-54af345ef794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:08.469913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.469891    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-xks-ca" pod "eccf5bbe-61df-4855-b55c-54af345ef794" (UID: "eccf5bbe-61df-4855-b55c-54af345ef794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:08.469997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.469974    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eccf5bbe-61df-4855-b55c-54af345ef794-kube-api-access-d8dj8" pod "eccf5bbe-61df-4855-b55c-54af345ef794" (UID: "eccf5bbe-61df-4855-b55c-54af345ef794"). InnerVolumeSpecName "kube-api-access-d8dj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:08.569180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.569144    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8dj8\" (UniqueName: \"kubernetes.io/projected/eccf5bbe-61df-4855-b55c-54af345ef794-kube-api-access-d8dj8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:08.569180 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.569178    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:08.569405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:08.569188    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eccf5bbe-61df-4855-b55c-54af345ef794-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:09.248864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:09.248835    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"
Oct 30 19:52:09.249050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:09.248833    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w" event={"ID":"eccf5bbe-61df-4855-b55c-54af345ef794","Type":"ContainerDied","Data":"1a298b379cbd11e33b973d3ecf075d55b71b14feeedaa6198b94cf5c27f4ae63"}
Oct 30 19:52:09.249050 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:09.248968    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a298b379cbd11e33b973d3ecf075d55b71b14feeedaa6198b94cf5c27f4ae63"
Oct 30 19:52:13.639802 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:52:13.639758    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:52:15.368128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:15.368083    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:52:19.727759 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:19.727715    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:52:25.437982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.437948    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"]
Oct 30 19:52:25.438484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.438455    6777 cpu_manager.go:414] "containerMap: removing container" podUID="eccf5bbe-61df-4855-b55c-54af345ef794" containerName="revoke-keys"
Oct 30 19:52:25.438484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.438469    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eccf5bbe-61df-4855-b55c-54af345ef794" containerName="revoke-keys"
Oct 30 19:52:25.438580 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.438555    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="eccf5bbe-61df-4855-b55c-54af345ef794" containerName="revoke-keys"
Oct 30 19:52:25.442172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.442145    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.449690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.449654    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:52:25.449895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.449770    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:52:25.449895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.449799    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:25.449895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.449841    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:52:25.450818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.450782    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"]
Oct 30 19:52:25.598239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.598200    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p2v5b\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.598475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.598398    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp865\" (UniqueName: \"kubernetes.io/projected/bfe40a99-8179-4deb-b3f5-299f4fafdd62-kube-api-access-xp865\") pod \"maas-api-revoke-sub-ae594979-p2v5b\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.598475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.598454    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p2v5b\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.699175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.699042    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p2v5b\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.699344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.699196    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp865\" (UniqueName: \"kubernetes.io/projected/bfe40a99-8179-4deb-b3f5-299f4fafdd62-kube-api-access-xp865\") pod \"maas-api-revoke-sub-ae594979-p2v5b\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.699344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.699224    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p2v5b\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.699836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.699800    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p2v5b\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.701649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.701612    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p2v5b\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.712207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.712180    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp865\" (UniqueName: \"kubernetes.io/projected/bfe40a99-8179-4deb-b3f5-299f4fafdd62-kube-api-access-xp865\") pod \"maas-api-revoke-sub-ae594979-p2v5b\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:25.759512 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:25.759474    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:26.035316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:26.035287    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"]
Oct 30 19:52:26.037321 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:52:26.037288    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe40a99_8179_4deb_b3f5_299f4fafdd62.slice/crio-32e18f6df3365646da601da467b1e166c76709526f1fced92b90cacffa449d86 WatchSource:0}: Error finding container 32e18f6df3365646da601da467b1e166c76709526f1fced92b90cacffa449d86: Status 404 returned error can't find the container with id 32e18f6df3365646da601da467b1e166c76709526f1fced92b90cacffa449d86
Oct 30 19:52:26.367805 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:26.367770    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b" event={"ID":"bfe40a99-8179-4deb-b3f5-299f4fafdd62","Type":"ContainerStarted","Data":"071c276205a49794889050f4a7f4d3d295c2bf9f5cc2d60419f1d1486e7c46ce"}
Oct 30 19:52:26.367934 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:26.367825    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b" event={"ID":"bfe40a99-8179-4deb-b3f5-299f4fafdd62","Type":"ContainerStarted","Data":"32e18f6df3365646da601da467b1e166c76709526f1fced92b90cacffa449d86"}
Oct 30 19:52:26.387792 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:26.387736    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b" podStartSLOduration=1.387721268 podStartE2EDuration="1.387721268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:26.386926045 +0000 UTC m=+2728.383841430" watchObservedRunningTime="2026-10-30 19:52:26.387721268 +0000 UTC m=+2728.384636651"
Oct 30 19:52:27.375313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:27.375273    6777 generic.go:356] "Generic (PLEG): container finished" podID="bfe40a99-8179-4deb-b3f5-299f4fafdd62" containerID="071c276205a49794889050f4a7f4d3d295c2bf9f5cc2d60419f1d1486e7c46ce" exitCode=0
Oct 30 19:52:27.375710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:27.375345    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b" event={"ID":"bfe40a99-8179-4deb-b3f5-299f4fafdd62","Type":"ContainerDied","Data":"071c276205a49794889050f4a7f4d3d295c2bf9f5cc2d60419f1d1486e7c46ce"}
Oct 30 19:52:28.519276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.519245    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:28.631516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.631419    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-openshift-ca\") pod \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") "
Oct 30 19:52:28.631516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.631505    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe40a99-8179-4deb-b3f5-299f4fafdd62-kube-api-access-xp865\" (UniqueName: \"kubernetes.io/projected/bfe40a99-8179-4deb-b3f5-299f4fafdd62-kube-api-access-xp865\") pod \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") "
Oct 30 19:52:28.631735 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.631527    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-xks-ca\") pod \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\" (UID: \"bfe40a99-8179-4deb-b3f5-299f4fafdd62\") "
Oct 30 19:52:28.631807 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.631780    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-openshift-ca" pod "bfe40a99-8179-4deb-b3f5-299f4fafdd62" (UID: "bfe40a99-8179-4deb-b3f5-299f4fafdd62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:28.633597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.633565    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe40a99-8179-4deb-b3f5-299f4fafdd62-kube-api-access-xp865" pod "bfe40a99-8179-4deb-b3f5-299f4fafdd62" (UID: "bfe40a99-8179-4deb-b3f5-299f4fafdd62"). InnerVolumeSpecName "kube-api-access-xp865". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:28.633729 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.633622    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-xks-ca" pod "bfe40a99-8179-4deb-b3f5-299f4fafdd62" (UID: "bfe40a99-8179-4deb-b3f5-299f4fafdd62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:28.733124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.733045    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp865\" (UniqueName: \"kubernetes.io/projected/bfe40a99-8179-4deb-b3f5-299f4fafdd62-kube-api-access-xp865\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:28.733124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.733121    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:28.733348 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:28.733137    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe40a99-8179-4deb-b3f5-299f4fafdd62-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:29.387030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:29.386997    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"
Oct 30 19:52:29.387030 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:29.387009    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b" event={"ID":"bfe40a99-8179-4deb-b3f5-299f4fafdd62","Type":"ContainerDied","Data":"32e18f6df3365646da601da467b1e166c76709526f1fced92b90cacffa449d86"}
Oct 30 19:52:29.387283 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:29.387073    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32e18f6df3365646da601da467b1e166c76709526f1fced92b90cacffa449d86"
Oct 30 19:52:36.172425 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:36.172387    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:52:39.429797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:39.429749    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:52:43.640044 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:52:43.640002    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:52:43.691715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.691673    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"]
Oct 30 19:52:43.692401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.692378    6777 cpu_manager.go:414] "containerMap: removing container" podUID="bfe40a99-8179-4deb-b3f5-299f4fafdd62" containerName="revoke-keys"
Oct 30 19:52:43.692458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.692404    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe40a99-8179-4deb-b3f5-299f4fafdd62" containerName="revoke-keys"
Oct 30 19:52:43.692605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.692593    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe40a99-8179-4deb-b3f5-299f4fafdd62" containerName="revoke-keys"
Oct 30 19:52:43.696800 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.696781    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:43.702678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.702464    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:43.702678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.702553    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:52:43.702678 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.702646    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:52:43.702962 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.702757    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:52:43.703190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.703160    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"]
Oct 30 19:52:43.728300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.728262    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zzjc9\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:43.728300 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.728303    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zzjc9\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:43.728597 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.728452    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszsr\" (UniqueName: \"kubernetes.io/projected/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-kube-api-access-gszsr\") pod \"maas-api-revoke-sub-a4aef7d7-zzjc9\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:43.829216 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.829162    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zzjc9\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:43.829427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.829323    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszsr\" (UniqueName: \"kubernetes.io/projected/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-kube-api-access-gszsr\") pod \"maas-api-revoke-sub-a4aef7d7-zzjc9\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:43.829471 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.829435    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zzjc9\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:43.830069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.830037    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zzjc9\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:43.831661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.831637    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zzjc9\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:43.842523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:43.842497    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszsr\" (UniqueName: \"kubernetes.io/projected/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-kube-api-access-gszsr\") pod \"maas-api-revoke-sub-a4aef7d7-zzjc9\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:44.015682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:44.015590    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:44.290492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:44.290465    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"]
Oct 30 19:52:44.292826 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:52:44.292788    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaffb6b1_9d3b_4868_95c6_f7b80674ec77.slice/crio-190204a85525dfac7f1b8eeed43540248ef3866d40007333dafca3e9e6a19ea1 WatchSource:0}: Error finding container 190204a85525dfac7f1b8eeed43540248ef3866d40007333dafca3e9e6a19ea1: Status 404 returned error can't find the container with id 190204a85525dfac7f1b8eeed43540248ef3866d40007333dafca3e9e6a19ea1
Oct 30 19:52:44.613829 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:44.613790    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9" event={"ID":"eaffb6b1-9d3b-4868-95c6-f7b80674ec77","Type":"ContainerStarted","Data":"daa4106a021ddbd89bbf51451bff1cbfca632ee5c7a8e0f874e9e575a82c37ac"}
Oct 30 19:52:44.614042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:44.613846    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9" event={"ID":"eaffb6b1-9d3b-4868-95c6-f7b80674ec77","Type":"ContainerStarted","Data":"190204a85525dfac7f1b8eeed43540248ef3866d40007333dafca3e9e6a19ea1"}
Oct 30 19:52:44.634272 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:44.634221    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9" podStartSLOduration=1.6342067550000001 podStartE2EDuration="1.634206755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:44.633662709 +0000 UTC m=+2746.630578091" watchObservedRunningTime="2026-10-30 19:52:44.634206755 +0000 UTC m=+2746.631122133"
Oct 30 19:52:45.621144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:45.621108    6777 generic.go:356] "Generic (PLEG): container finished" podID="eaffb6b1-9d3b-4868-95c6-f7b80674ec77" containerID="daa4106a021ddbd89bbf51451bff1cbfca632ee5c7a8e0f874e9e575a82c37ac" exitCode=0
Oct 30 19:52:45.621523 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:45.621190    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9" event={"ID":"eaffb6b1-9d3b-4868-95c6-f7b80674ec77","Type":"ContainerDied","Data":"daa4106a021ddbd89bbf51451bff1cbfca632ee5c7a8e0f874e9e575a82c37ac"}
Oct 30 19:52:46.759261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.759230    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:46.862152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.862112    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-xks-ca\") pod \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") "
Oct 30 19:52:46.862152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.862162    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-kube-api-access-gszsr\" (UniqueName: \"kubernetes.io/projected/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-kube-api-access-gszsr\") pod \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") "
Oct 30 19:52:46.862438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.862221    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-openshift-ca\") pod \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\" (UID: \"eaffb6b1-9d3b-4868-95c6-f7b80674ec77\") "
Oct 30 19:52:46.862681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.862653    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-openshift-ca" pod "eaffb6b1-9d3b-4868-95c6-f7b80674ec77" (UID: "eaffb6b1-9d3b-4868-95c6-f7b80674ec77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:46.864348 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.864323    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-xks-ca" pod "eaffb6b1-9d3b-4868-95c6-f7b80674ec77" (UID: "eaffb6b1-9d3b-4868-95c6-f7b80674ec77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:46.864460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.864391    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-kube-api-access-gszsr" pod "eaffb6b1-9d3b-4868-95c6-f7b80674ec77" (UID: "eaffb6b1-9d3b-4868-95c6-f7b80674ec77"). InnerVolumeSpecName "kube-api-access-gszsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:46.963427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.963333    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:46.963427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.963370    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:46.963427 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:46.963384    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gszsr\" (UniqueName: \"kubernetes.io/projected/eaffb6b1-9d3b-4868-95c6-f7b80674ec77-kube-api-access-gszsr\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:47.633505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:47.633465    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9" event={"ID":"eaffb6b1-9d3b-4868-95c6-f7b80674ec77","Type":"ContainerDied","Data":"190204a85525dfac7f1b8eeed43540248ef3866d40007333dafca3e9e6a19ea1"}
Oct 30 19:52:47.633505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:47.633501    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"
Oct 30 19:52:47.633731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:47.633531    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="190204a85525dfac7f1b8eeed43540248ef3866d40007333dafca3e9e6a19ea1"
Oct 30 19:52:47.692190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:47.692158    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"]
Oct 30 19:52:47.692472 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:47.692442    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9" podUID="61950875-cf6e-4bea-8e3c-ef6a6c6c4499" containerName="discovery" containerID="cri-o://45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605" gracePeriod=30
Oct 30 19:52:47.949090 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:47.949033    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:52:48.075374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.075337    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-local-certs\") pod \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") "
Oct 30 19:52:48.075575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.075422    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-ca-configmap\") pod \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") "
Oct 30 19:52:48.075575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.075448    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-dns-cert\") pod \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") "
Oct 30 19:52:48.075575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.075470    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-kube-api-access-dwfn2\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-kube-api-access-dwfn2\") pod \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") "
Oct 30 19:52:48.075575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.075503    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-cacerts\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-cacerts\") pod \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") "
Oct 30 19:52:48.075575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.075550    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-token\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-token\") pod \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") "
Oct 30 19:52:48.075860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.075603    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-kubeconfig\") pod \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\" (UID: \"61950875-cf6e-4bea-8e3c-ef6a6c6c4499\") "
Oct 30 19:52:48.075955 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.075917    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-ca-configmap" pod "61950875-cf6e-4bea-8e3c-ef6a6c6c4499" (UID: "61950875-cf6e-4bea-8e3c-ef6a6c6c4499"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:48.077912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.077880    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-dns-cert" pod "61950875-cf6e-4bea-8e3c-ef6a6c6c4499" (UID: "61950875-cf6e-4bea-8e3c-ef6a6c6c4499"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:48.078037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.077933    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-token" pod "61950875-cf6e-4bea-8e3c-ef6a6c6c4499" (UID: "61950875-cf6e-4bea-8e3c-ef6a6c6c4499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:48.078224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.078194    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-kube-api-access-dwfn2" pod "61950875-cf6e-4bea-8e3c-ef6a6c6c4499" (UID: "61950875-cf6e-4bea-8e3c-ef6a6c6c4499"). InnerVolumeSpecName "kube-api-access-dwfn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:48.078315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.078256    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-local-certs" pod "61950875-cf6e-4bea-8e3c-ef6a6c6c4499" (UID: "61950875-cf6e-4bea-8e3c-ef6a6c6c4499"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:48.078315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.078260    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-cacerts" pod "61950875-cf6e-4bea-8e3c-ef6a6c6c4499" (UID: "61950875-cf6e-4bea-8e3c-ef6a6c6c4499"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:48.078315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.078293    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-kubeconfig" pod "61950875-cf6e-4bea-8e3c-ef6a6c6c4499" (UID: "61950875-cf6e-4bea-8e3c-ef6a6c6c4499"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:48.176859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.176752    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:48.176859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.176793    6777 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-kubeconfig\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:48.176859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.176808    6777 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-local-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:48.176859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.176825    6777 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-ca-configmap\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:48.176859 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.176838    6777 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-istio-csr-dns-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:48.177242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.176949    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwfn2\" (UniqueName: \"kubernetes.io/projected/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-kube-api-access-dwfn2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:48.177242 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.176973    6777 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61950875-cf6e-4bea-8e3c-ef6a6c6c4499-cacerts\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:48.639959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.639915    6777 generic.go:356] "Generic (PLEG): container finished" podID="61950875-cf6e-4bea-8e3c-ef6a6c6c4499" containerID="45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605" exitCode=0
Oct 30 19:52:48.639959 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.639960    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9" event={"ID":"61950875-cf6e-4bea-8e3c-ef6a6c6c4499","Type":"ContainerDied","Data":"45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605"}
Oct 30 19:52:48.640260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.639971    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"
Oct 30 19:52:48.640260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.639999    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9" event={"ID":"61950875-cf6e-4bea-8e3c-ef6a6c6c4499","Type":"ContainerDied","Data":"52820a57af8db3b0f4252b2d38727a055b954949e7eceb246048afeafe85aea5"}
Oct 30 19:52:48.640260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.640018    6777 scope.go:122] "RemoveContainer" containerID="45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605"
Oct 30 19:52:48.650576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.650556    6777 scope.go:122] "RemoveContainer" containerID="45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605"
Oct 30 19:52:48.650855 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:52:48.650833    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605\": container with ID starting with 45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605 not found: ID does not exist" containerID="45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605"
Oct 30 19:52:48.650964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.650860    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605"} err="failed to get container status \"45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605\": rpc error: code = NotFound desc = could not find container \"45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605\": container with ID starting with 45028d8aa2e339ab1d9a4087e3e8b1e7639331744a168eb6a64c6ac17e1a9605 not found: ID does not exist"
Oct 30 19:52:48.662023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.661992    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"]
Oct 30 19:52:48.665099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:48.665067    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk2r9"]
Oct 30 19:52:50.111405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:50.111371    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61950875-cf6e-4bea-8e3c-ef6a6c6c4499" path="/var/lib/kubelet/pods/61950875-cf6e-4bea-8e3c-ef6a6c6c4499/volumes"
Oct 30 19:52:54.443664 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:54.443621    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:52:58.793083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:52:58.792495    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:53:02.402828 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.402786    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"]
Oct 30 19:53:02.403511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.403481    6777 cpu_manager.go:414] "containerMap: removing container" podUID="eaffb6b1-9d3b-4868-95c6-f7b80674ec77" containerName="revoke-keys"
Oct 30 19:53:02.403511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.403509    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaffb6b1-9d3b-4868-95c6-f7b80674ec77" containerName="revoke-keys"
Oct 30 19:53:02.403707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.403530    6777 cpu_manager.go:414] "containerMap: removing container" podUID="61950875-cf6e-4bea-8e3c-ef6a6c6c4499" containerName="discovery"
Oct 30 19:53:02.403707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.403537    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61950875-cf6e-4bea-8e3c-ef6a6c6c4499" containerName="discovery"
Oct 30 19:53:02.403707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.403665    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaffb6b1-9d3b-4868-95c6-f7b80674ec77" containerName="revoke-keys"
Oct 30 19:53:02.403707 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.403675    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="61950875-cf6e-4bea-8e3c-ef6a6c6c4499" containerName="discovery"
Oct 30 19:53:02.406913 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.406895    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.414010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.413980    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:53:02.414254 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.414027    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:02.414359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.414134    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:02.415413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.415367    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:02.416989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.416962    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"]
Oct 30 19:53:02.550897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.550857    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgv2\" (UniqueName: \"kubernetes.io/projected/c5fe8684-0a04-4918-b50c-72cfd54b7db4-kube-api-access-nkgv2\") pod \"maas-api-revoke-sub-79805ca9-2dlq2\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.551166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.550909    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2dlq2\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.551166 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.551136    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2dlq2\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.652349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.652296    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgv2\" (UniqueName: \"kubernetes.io/projected/c5fe8684-0a04-4918-b50c-72cfd54b7db4-kube-api-access-nkgv2\") pod \"maas-api-revoke-sub-79805ca9-2dlq2\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.652576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.652372    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2dlq2\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.652576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.652503    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2dlq2\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.653249 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.653192    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2dlq2\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.654825 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.654805    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2dlq2\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.665414 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.665387    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgv2\" (UniqueName: \"kubernetes.io/projected/c5fe8684-0a04-4918-b50c-72cfd54b7db4-kube-api-access-nkgv2\") pod \"maas-api-revoke-sub-79805ca9-2dlq2\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:02.723282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:02.723245    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:03.001010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:03.000984    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"]
Oct 30 19:53:03.003751 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:53:03.003725    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fe8684_0a04_4918_b50c_72cfd54b7db4.slice/crio-5f58e9f53c2daaa2446c83ac8f5c54e9e3f3268643f3584c1661703b4f8e1365 WatchSource:0}: Error finding container 5f58e9f53c2daaa2446c83ac8f5c54e9e3f3268643f3584c1661703b4f8e1365: Status 404 returned error can't find the container with id 5f58e9f53c2daaa2446c83ac8f5c54e9e3f3268643f3584c1661703b4f8e1365
Oct 30 19:53:03.757600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:03.757565    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2" event={"ID":"c5fe8684-0a04-4918-b50c-72cfd54b7db4","Type":"ContainerStarted","Data":"6be4606fbb837fa0989b923d1abc1934a7010a82425620fc62abec558882dca1"}
Oct 30 19:53:03.758093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:03.757621    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2" event={"ID":"c5fe8684-0a04-4918-b50c-72cfd54b7db4","Type":"ContainerStarted","Data":"5f58e9f53c2daaa2446c83ac8f5c54e9e3f3268643f3584c1661703b4f8e1365"}
Oct 30 19:53:03.779217 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:03.779155    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2" podStartSLOduration=1.7791386569999998 podStartE2EDuration="1.779138657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:03.77631767 +0000 UTC m=+2765.773233053" watchObservedRunningTime="2026-10-30 19:53:03.779138657 +0000 UTC m=+2765.776054040"
Oct 30 19:53:04.764922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:04.764883    6777 generic.go:356] "Generic (PLEG): container finished" podID="c5fe8684-0a04-4918-b50c-72cfd54b7db4" containerID="6be4606fbb837fa0989b923d1abc1934a7010a82425620fc62abec558882dca1" exitCode=0
Oct 30 19:53:04.765346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:04.764966    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2" event={"ID":"c5fe8684-0a04-4918-b50c-72cfd54b7db4","Type":"ContainerDied","Data":"6be4606fbb837fa0989b923d1abc1934a7010a82425620fc62abec558882dca1"}
Oct 30 19:53:05.912456 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:05.912428    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:05.989168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:05.989132    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-xks-ca\") pod \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") "
Oct 30 19:53:05.989383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:05.989224    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-openshift-ca\") pod \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") "
Oct 30 19:53:05.989383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:05.989265    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fe8684-0a04-4918-b50c-72cfd54b7db4-kube-api-access-nkgv2\" (UniqueName: \"kubernetes.io/projected/c5fe8684-0a04-4918-b50c-72cfd54b7db4-kube-api-access-nkgv2\") pod \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\" (UID: \"c5fe8684-0a04-4918-b50c-72cfd54b7db4\") "
Oct 30 19:53:05.989638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:05.989613    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-openshift-ca" pod "c5fe8684-0a04-4918-b50c-72cfd54b7db4" (UID: "c5fe8684-0a04-4918-b50c-72cfd54b7db4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:05.991320 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:05.991292    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-xks-ca" pod "c5fe8684-0a04-4918-b50c-72cfd54b7db4" (UID: "c5fe8684-0a04-4918-b50c-72cfd54b7db4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:05.991436 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:05.991347    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fe8684-0a04-4918-b50c-72cfd54b7db4-kube-api-access-nkgv2" pod "c5fe8684-0a04-4918-b50c-72cfd54b7db4" (UID: "c5fe8684-0a04-4918-b50c-72cfd54b7db4"). InnerVolumeSpecName "kube-api-access-nkgv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:06.091043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:06.091000    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.091043 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:06.091041    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe8684-0a04-4918-b50c-72cfd54b7db4-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.091322 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:06.091083    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgv2\" (UniqueName: \"kubernetes.io/projected/c5fe8684-0a04-4918-b50c-72cfd54b7db4-kube-api-access-nkgv2\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:06.776809 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:06.776762    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2" event={"ID":"c5fe8684-0a04-4918-b50c-72cfd54b7db4","Type":"ContainerDied","Data":"5f58e9f53c2daaa2446c83ac8f5c54e9e3f3268643f3584c1661703b4f8e1365"}
Oct 30 19:53:06.776989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:06.776822    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f58e9f53c2daaa2446c83ac8f5c54e9e3f3268643f3584c1661703b4f8e1365"
Oct 30 19:53:06.776989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:06.776773    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"
Oct 30 19:53:13.146964 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:13.146928    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:53:13.642662 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:53:13.642602    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:53:16.343481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:16.343422    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:53:17.431385 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:17.431333    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:53:17.432111 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:17.432046    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:20.199148 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.199104    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"]
Oct 30 19:53:20.199713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.199681    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c5fe8684-0a04-4918-b50c-72cfd54b7db4" containerName="revoke-keys"
Oct 30 19:53:20.199713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.199699    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fe8684-0a04-4918-b50c-72cfd54b7db4" containerName="revoke-keys"
Oct 30 19:53:20.199850 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.199813    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fe8684-0a04-4918-b50c-72cfd54b7db4" containerName="revoke-keys"
Oct 30 19:53:20.204527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.204502    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.213159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.213091    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:53:20.213335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.213154    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:20.213335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.213160    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:20.213335 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.213237    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:20.213534 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.213427    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"]
Oct 30 19:53:20.389073 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.389028    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-662xx\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.389268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.389154    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlt58\" (UniqueName: \"kubernetes.io/projected/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-kube-api-access-xlt58\") pod \"maas-api-revoke-sub-0b7e8edb-662xx\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.389268 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.389239    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-662xx\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.490400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.490296    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-662xx\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.490400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.490345    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlt58\" (UniqueName: \"kubernetes.io/projected/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-kube-api-access-xlt58\") pod \"maas-api-revoke-sub-0b7e8edb-662xx\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.490400 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.490387    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-662xx\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.490956 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.490929    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-662xx\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.492927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.492904    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-662xx\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.503015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.502982    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlt58\" (UniqueName: \"kubernetes.io/projected/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-kube-api-access-xlt58\") pod \"maas-api-revoke-sub-0b7e8edb-662xx\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.519784 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.519749    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:20.794094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.794035    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"]
Oct 30 19:53:20.795958 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:53:20.795930    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3472bece_4e64_4a0d_bab7_b7bb22b6f0ec.slice/crio-cabfa0c2015c3e31718c8adf896e2deab969a998f9b0732351644f7fd6813efa WatchSource:0}: Error finding container cabfa0c2015c3e31718c8adf896e2deab969a998f9b0732351644f7fd6813efa: Status 404 returned error can't find the container with id cabfa0c2015c3e31718c8adf896e2deab969a998f9b0732351644f7fd6813efa
Oct 30 19:53:20.883252 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:20.883214    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx" event={"ID":"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec","Type":"ContainerStarted","Data":"cabfa0c2015c3e31718c8adf896e2deab969a998f9b0732351644f7fd6813efa"}
Oct 30 19:53:21.893646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:21.892992    6777 generic.go:356] "Generic (PLEG): container finished" podID="3472bece-4e64-4a0d-bab7-b7bb22b6f0ec" containerID="dce7fc87b4d8637dff44f36b4471805420e02812302b4a8be6b1019851715400" exitCode=0
Oct 30 19:53:21.893646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:21.893178    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx" event={"ID":"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec","Type":"ContainerDied","Data":"dce7fc87b4d8637dff44f36b4471805420e02812302b4a8be6b1019851715400"}
Oct 30 19:53:23.031648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.031606    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"]
Oct 30 19:53:23.037796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.037762    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"]
Oct 30 19:53:23.040006 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.039978    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f176a999-5q76s"]
Oct 30 19:53:23.042125 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.042102    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8b9bed-wcrsd"]
Oct 30 19:53:23.044007 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.043990    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:23.117458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.117427    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-xks-ca\") pod \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") "
Oct 30 19:53:23.117686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.117473    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-kube-api-access-xlt58\" (UniqueName: \"kubernetes.io/projected/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-kube-api-access-xlt58\") pod \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") "
Oct 30 19:53:23.117686 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.117532    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-openshift-ca\") pod \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\" (UID: \"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec\") "
Oct 30 19:53:23.118001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.117966    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-openshift-ca" pod "3472bece-4e64-4a0d-bab7-b7bb22b6f0ec" (UID: "3472bece-4e64-4a0d-bab7-b7bb22b6f0ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:23.118194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.118153    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:23.119869 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.119842    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-xks-ca" pod "3472bece-4e64-4a0d-bab7-b7bb22b6f0ec" (UID: "3472bece-4e64-4a0d-bab7-b7bb22b6f0ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:23.119869 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.119860    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-kube-api-access-xlt58" pod "3472bece-4e64-4a0d-bab7-b7bb22b6f0ec" (UID: "3472bece-4e64-4a0d-bab7-b7bb22b6f0ec"). InnerVolumeSpecName "kube-api-access-xlt58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:23.219340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.219236    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:23.219340 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.219276    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlt58\" (UniqueName: \"kubernetes.io/projected/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec-kube-api-access-xlt58\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:23.906276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.906245    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"
Oct 30 19:53:23.906561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.906282    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx" event={"ID":"3472bece-4e64-4a0d-bab7-b7bb22b6f0ec","Type":"ContainerDied","Data":"cabfa0c2015c3e31718c8adf896e2deab969a998f9b0732351644f7fd6813efa"}
Oct 30 19:53:23.906561 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:23.906333    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cabfa0c2015c3e31718c8adf896e2deab969a998f9b0732351644f7fd6813efa"
Oct 30 19:53:24.110881 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:24.110846    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd9d42f-365e-451b-8d03-daca355eb025" path="/var/lib/kubelet/pods/cbd9d42f-365e-451b-8d03-daca355eb025/volumes"
Oct 30 19:53:24.111366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:24.111350    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df954591-84c1-4e39-89a9-2eb186ed69fa" path="/var/lib/kubelet/pods/df954591-84c1-4e39-89a9-2eb186ed69fa/volumes"
Oct 30 19:53:30.868040 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.867994    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"]
Oct 30 19:53:30.868570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.868534    6777 cpu_manager.go:414] "containerMap: removing container" podUID="3472bece-4e64-4a0d-bab7-b7bb22b6f0ec" containerName="revoke-keys"
Oct 30 19:53:30.868570 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.868551    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3472bece-4e64-4a0d-bab7-b7bb22b6f0ec" containerName="revoke-keys"
Oct 30 19:53:30.868720 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.868705    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="3472bece-4e64-4a0d-bab7-b7bb22b6f0ec" containerName="revoke-keys"
Oct 30 19:53:30.871891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.871869    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:30.881312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.881284    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:30.881468 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.881323    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:30.881522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.881491    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:30.881585 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.881570    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:53:30.881837 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.881807    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"]
Oct 30 19:53:30.899143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.899108    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwj22\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:30.899347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.899274    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwj22\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:30.899347 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.899331    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752zk\" (UniqueName: \"kubernetes.io/projected/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-kube-api-access-752zk\") pod \"maas-api-revoke-sub-3d4156f1-cwj22\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:30.943094 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:30.943039    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:53:31.000685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.000639    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwj22\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:31.000908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.000711    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752zk\" (UniqueName: \"kubernetes.io/projected/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-kube-api-access-752zk\") pod \"maas-api-revoke-sub-3d4156f1-cwj22\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:31.000908 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.000796    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwj22\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:31.001438 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.001410    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwj22\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:31.003491 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.003468    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cwj22\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:31.014182 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.014154    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-752zk\" (UniqueName: \"kubernetes.io/projected/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-kube-api-access-752zk\") pod \"maas-api-revoke-sub-3d4156f1-cwj22\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:31.188978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.188887    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:31.467743 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.467712    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"]
Oct 30 19:53:31.469749 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:53:31.469722    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7488a53a_0a1a_4a10_b0f1_e76b1460c10a.slice/crio-1013e02a0f8b41b223b78aa77d4e887783b3105c8e7e12a65a9de915aefea2e3 WatchSource:0}: Error finding container 1013e02a0f8b41b223b78aa77d4e887783b3105c8e7e12a65a9de915aefea2e3: Status 404 returned error can't find the container with id 1013e02a0f8b41b223b78aa77d4e887783b3105c8e7e12a65a9de915aefea2e3
Oct 30 19:53:31.952860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.952825    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22" event={"ID":"7488a53a-0a1a-4a10-b0f1-e76b1460c10a","Type":"ContainerStarted","Data":"2e557200d7e5768a730c2dc82f2530828283b21f15aa5f1a8071cac473b15427"}
Oct 30 19:53:31.953342 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.952877    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22" event={"ID":"7488a53a-0a1a-4a10-b0f1-e76b1460c10a","Type":"ContainerStarted","Data":"1013e02a0f8b41b223b78aa77d4e887783b3105c8e7e12a65a9de915aefea2e3"}
Oct 30 19:53:31.974173 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:31.974110    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22" podStartSLOduration=1.974095552 podStartE2EDuration="1.974095552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:31.972010358 +0000 UTC m=+2793.968925746" watchObservedRunningTime="2026-10-30 19:53:31.974095552 +0000 UTC m=+2793.971010927"
Oct 30 19:53:32.961124 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:32.961086    6777 generic.go:356] "Generic (PLEG): container finished" podID="7488a53a-0a1a-4a10-b0f1-e76b1460c10a" containerID="2e557200d7e5768a730c2dc82f2530828283b21f15aa5f1a8071cac473b15427" exitCode=0
Oct 30 19:53:32.961563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:32.961187    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22" event={"ID":"7488a53a-0a1a-4a10-b0f1-e76b1460c10a","Type":"ContainerDied","Data":"2e557200d7e5768a730c2dc82f2530828283b21f15aa5f1a8071cac473b15427"}
Oct 30 19:53:34.106858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.106830    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:34.233231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.233134    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-openshift-ca\") pod \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") "
Oct 30 19:53:34.233231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.233196    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-xks-ca\") pod \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") "
Oct 30 19:53:34.233439 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.233263    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-kube-api-access-752zk\" (UniqueName: \"kubernetes.io/projected/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-kube-api-access-752zk\") pod \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\" (UID: \"7488a53a-0a1a-4a10-b0f1-e76b1460c10a\") "
Oct 30 19:53:34.233538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.233515    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-openshift-ca" pod "7488a53a-0a1a-4a10-b0f1-e76b1460c10a" (UID: "7488a53a-0a1a-4a10-b0f1-e76b1460c10a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:34.233661 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.233637    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:34.235431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.235400    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-xks-ca" pod "7488a53a-0a1a-4a10-b0f1-e76b1460c10a" (UID: "7488a53a-0a1a-4a10-b0f1-e76b1460c10a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:34.235543 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.235429    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-kube-api-access-752zk" pod "7488a53a-0a1a-4a10-b0f1-e76b1460c10a" (UID: "7488a53a-0a1a-4a10-b0f1-e76b1460c10a"). InnerVolumeSpecName "kube-api-access-752zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:34.335327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.335282    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:34.335327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.335319    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-752zk\" (UniqueName: \"kubernetes.io/projected/7488a53a-0a1a-4a10-b0f1-e76b1460c10a-kube-api-access-752zk\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:34.973922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.973881    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22" event={"ID":"7488a53a-0a1a-4a10-b0f1-e76b1460c10a","Type":"ContainerDied","Data":"1013e02a0f8b41b223b78aa77d4e887783b3105c8e7e12a65a9de915aefea2e3"}
Oct 30 19:53:34.973922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.973922    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"
Oct 30 19:53:34.974183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:34.973945    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1013e02a0f8b41b223b78aa77d4e887783b3105c8e7e12a65a9de915aefea2e3"
Oct 30 19:53:41.566480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:41.566403    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:53:43.748633 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:53:43.748557    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:53:45.360029 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:45.359994    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:53:48.975321 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:48.975285    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"]
Oct 30 19:53:48.981181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:48.981155    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cwj22"]
Oct 30 19:53:48.999224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:48.999195    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"]
Oct 30 19:53:48.999708 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:48.999688    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7488a53a-0a1a-4a10-b0f1-e76b1460c10a" containerName="revoke-keys"
Oct 30 19:53:48.999796 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:48.999710    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7488a53a-0a1a-4a10-b0f1-e76b1460c10a" containerName="revoke-keys"
Oct 30 19:53:48.999861 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:48.999825    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7488a53a-0a1a-4a10-b0f1-e76b1460c10a" containerName="revoke-keys"
Oct 30 19:53:49.003260 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.003242    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.010712 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.010691    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:49.010858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.010831    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:49.013032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.013010    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"]
Oct 30 19:53:49.047778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.047738    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:49.047948 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.047826    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:53:49.120858 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.120835    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cg6h\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.120989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.120890    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ml7\" (UniqueName: \"kubernetes.io/projected/44b58ddc-5b98-4505-86d0-c8f931d19caa-kube-api-access-m2ml7\") pod \"maas-api-revoke-sub-3d4156f1-7cg6h\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.121036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.120991    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cg6h\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.222092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.222050    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cg6h\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.222204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.222118    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ml7\" (UniqueName: \"kubernetes.io/projected/44b58ddc-5b98-4505-86d0-c8f931d19caa-kube-api-access-m2ml7\") pod \"maas-api-revoke-sub-3d4156f1-7cg6h\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.222204 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.222156    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cg6h\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.222633 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.222615    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cg6h\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.224327 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.224308    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cg6h\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.234738 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.234688    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ml7\" (UniqueName: \"kubernetes.io/projected/44b58ddc-5b98-4505-86d0-c8f931d19caa-kube-api-access-m2ml7\") pod \"maas-api-revoke-sub-3d4156f1-7cg6h\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.318386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.318365    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:49.589394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:49.589373    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"]
Oct 30 19:53:50.110986 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:50.110921    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7488a53a-0a1a-4a10-b0f1-e76b1460c10a" path="/var/lib/kubelet/pods/7488a53a-0a1a-4a10-b0f1-e76b1460c10a/volumes"
Oct 30 19:53:50.115405 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:50.115378    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h" event={"ID":"44b58ddc-5b98-4505-86d0-c8f931d19caa","Type":"ContainerStarted","Data":"7739e3baf7828c48346fa43512457183932575369d0d5c0c84ed9ffc36b4c466"}
Oct 30 19:53:50.115516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:50.115432    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h" event={"ID":"44b58ddc-5b98-4505-86d0-c8f931d19caa","Type":"ContainerStarted","Data":"6eb65318ea66fdcb7ad7b416fb6faf8e4b8770fcf736044bd37bfcb20fe71a7a"}
Oct 30 19:53:50.137378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:50.137337    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h" podStartSLOduration=2.137325166 podStartE2EDuration="2.137325166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:50.134560233 +0000 UTC m=+2812.131475618" watchObservedRunningTime="2026-10-30 19:53:50.137325166 +0000 UTC m=+2812.134240550"
Oct 30 19:53:51.122763 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:51.122727    6777 generic.go:356] "Generic (PLEG): container finished" podID="44b58ddc-5b98-4505-86d0-c8f931d19caa" containerID="7739e3baf7828c48346fa43512457183932575369d0d5c0c84ed9ffc36b4c466" exitCode=0
Oct 30 19:53:51.123188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:51.122813    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h" event={"ID":"44b58ddc-5b98-4505-86d0-c8f931d19caa","Type":"ContainerDied","Data":"7739e3baf7828c48346fa43512457183932575369d0d5c0c84ed9ffc36b4c466"}
Oct 30 19:53:52.260542 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.260517    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:52.349930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.349902    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-xks-ca\") pod \"44b58ddc-5b98-4505-86d0-c8f931d19caa\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") "
Oct 30 19:53:52.350109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.349991    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b58ddc-5b98-4505-86d0-c8f931d19caa-kube-api-access-m2ml7\" (UniqueName: \"kubernetes.io/projected/44b58ddc-5b98-4505-86d0-c8f931d19caa-kube-api-access-m2ml7\") pod \"44b58ddc-5b98-4505-86d0-c8f931d19caa\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") "
Oct 30 19:53:52.350109 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.350030    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-openshift-ca\") pod \"44b58ddc-5b98-4505-86d0-c8f931d19caa\" (UID: \"44b58ddc-5b98-4505-86d0-c8f931d19caa\") "
Oct 30 19:53:52.350406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.350386    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-openshift-ca" pod "44b58ddc-5b98-4505-86d0-c8f931d19caa" (UID: "44b58ddc-5b98-4505-86d0-c8f931d19caa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:52.351877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.351857    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-xks-ca" pod "44b58ddc-5b98-4505-86d0-c8f931d19caa" (UID: "44b58ddc-5b98-4505-86d0-c8f931d19caa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:52.352001 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.351987    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b58ddc-5b98-4505-86d0-c8f931d19caa-kube-api-access-m2ml7" pod "44b58ddc-5b98-4505-86d0-c8f931d19caa" (UID: "44b58ddc-5b98-4505-86d0-c8f931d19caa"). InnerVolumeSpecName "kube-api-access-m2ml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:52.451113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.451034    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:52.451113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.451079    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b58ddc-5b98-4505-86d0-c8f931d19caa-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:52.451113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:52.451096    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2ml7\" (UniqueName: \"kubernetes.io/projected/44b58ddc-5b98-4505-86d0-c8f931d19caa-kube-api-access-m2ml7\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:53.134418 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:53.134397    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"
Oct 30 19:53:53.134568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:53.134444    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h" event={"ID":"44b58ddc-5b98-4505-86d0-c8f931d19caa","Type":"ContainerDied","Data":"6eb65318ea66fdcb7ad7b416fb6faf8e4b8770fcf736044bd37bfcb20fe71a7a"}
Oct 30 19:53:53.134568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:53.134486    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb65318ea66fdcb7ad7b416fb6faf8e4b8770fcf736044bd37bfcb20fe71a7a"
Oct 30 19:53:59.661286 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:53:59.661254    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:54:02.667584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:02.667554    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:54:03.025106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:03.025015    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"]
Oct 30 19:54:03.029778 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:03.029755    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb88e92-4nq4p"]
Oct 30 19:54:04.110357 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:04.110328    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ffe77ce-28ce-4512-a13b-f46175c14dc4" path="/var/lib/kubelet/pods/7ffe77ce-28ce-4512-a13b-f46175c14dc4/volumes"
Oct 30 19:54:06.568152 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:06.568121    6777 scope.go:122] "RemoveContainer" containerID="09d4be9b5a684e47842b72ccc623f2ebc7d111f824fc9f42e4f7e061afec5c0a"
Oct 30 19:54:09.639855 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.639821    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"]
Oct 30 19:54:09.643145 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.643122    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cg6h"]
Oct 30 19:54:09.665951 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.665928    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"]
Oct 30 19:54:09.666345 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.666330    6777 cpu_manager.go:414] "containerMap: removing container" podUID="44b58ddc-5b98-4505-86d0-c8f931d19caa" containerName="revoke-keys"
Oct 30 19:54:09.666399 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.666345    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b58ddc-5b98-4505-86d0-c8f931d19caa" containerName="revoke-keys"
Oct 30 19:54:09.666444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.666423    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b58ddc-5b98-4505-86d0-c8f931d19caa" containerName="revoke-keys"
Oct 30 19:54:09.670966 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.670946    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.676261 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.676194    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:54:09.676358 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.676288    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:54:09.677535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.677504    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:09.677782 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.677729    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:54:09.680042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.680022    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"]
Oct 30 19:54:09.799609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.799585    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dqg\" (UniqueName: \"kubernetes.io/projected/8120ce2e-a4af-4183-b215-cd1c31b31142-kube-api-access-97dqg\") pod \"maas-api-revoke-sub-3d4156f1-hz89w\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.799710 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.799623    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hz89w\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.799754 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.799707    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hz89w\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.900448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.900399    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hz89w\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.900448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.900445    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hz89w\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.900574 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.900499    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dqg\" (UniqueName: \"kubernetes.io/projected/8120ce2e-a4af-4183-b215-cd1c31b31142-kube-api-access-97dqg\") pod \"maas-api-revoke-sub-3d4156f1-hz89w\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.901105 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.901086    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hz89w\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.902645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.902627    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hz89w\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.913599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.913578    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dqg\" (UniqueName: \"kubernetes.io/projected/8120ce2e-a4af-4183-b215-cd1c31b31142-kube-api-access-97dqg\") pod \"maas-api-revoke-sub-3d4156f1-hz89w\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:09.985349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:09.985329    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:10.110579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:10.110546    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b58ddc-5b98-4505-86d0-c8f931d19caa" path="/var/lib/kubelet/pods/44b58ddc-5b98-4505-86d0-c8f931d19caa/volumes"
Oct 30 19:54:10.257087 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:10.257049    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"]
Oct 30 19:54:10.259210 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:54:10.259180    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8120ce2e_a4af_4183_b215_cd1c31b31142.slice/crio-9953cec0d5605016c0394031732222fd159a4944dc8f7b3c07c714604f26e4fb WatchSource:0}: Error finding container 9953cec0d5605016c0394031732222fd159a4944dc8f7b3c07c714604f26e4fb: Status 404 returned error can't find the container with id 9953cec0d5605016c0394031732222fd159a4944dc8f7b3c07c714604f26e4fb
Oct 30 19:54:11.229267 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:11.229236    6777 generic.go:356] "Generic (PLEG): container finished" podID="8120ce2e-a4af-4183-b215-cd1c31b31142" containerID="ca7f346cf226fb2edb8713316a12e62ef54b2e80d35360ba0f7f854c233779f4" exitCode=0
Oct 30 19:54:11.229586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:11.229314    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w" event={"ID":"8120ce2e-a4af-4183-b215-cd1c31b31142","Type":"ContainerDied","Data":"ca7f346cf226fb2edb8713316a12e62ef54b2e80d35360ba0f7f854c233779f4"}
Oct 30 19:54:11.229586 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:11.229359    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w" event={"ID":"8120ce2e-a4af-4183-b215-cd1c31b31142","Type":"ContainerStarted","Data":"9953cec0d5605016c0394031732222fd159a4944dc8f7b3c07c714604f26e4fb"}
Oct 30 19:54:12.359535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.359513    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:12.423135 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.423108    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8120ce2e-a4af-4183-b215-cd1c31b31142-kube-api-access-97dqg\" (UniqueName: \"kubernetes.io/projected/8120ce2e-a4af-4183-b215-cd1c31b31142-kube-api-access-97dqg\") pod \"8120ce2e-a4af-4183-b215-cd1c31b31142\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") "
Oct 30 19:54:12.423282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.423174    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-openshift-ca\") pod \"8120ce2e-a4af-4183-b215-cd1c31b31142\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") "
Oct 30 19:54:12.423282 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.423217    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-xks-ca\") pod \"8120ce2e-a4af-4183-b215-cd1c31b31142\" (UID: \"8120ce2e-a4af-4183-b215-cd1c31b31142\") "
Oct 30 19:54:12.423503 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.423480    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-openshift-ca" pod "8120ce2e-a4af-4183-b215-cd1c31b31142" (UID: "8120ce2e-a4af-4183-b215-cd1c31b31142"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:12.425093 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.425044    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-xks-ca" pod "8120ce2e-a4af-4183-b215-cd1c31b31142" (UID: "8120ce2e-a4af-4183-b215-cd1c31b31142"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:12.425199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.425104    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8120ce2e-a4af-4183-b215-cd1c31b31142-kube-api-access-97dqg" pod "8120ce2e-a4af-4183-b215-cd1c31b31142" (UID: "8120ce2e-a4af-4183-b215-cd1c31b31142"). InnerVolumeSpecName "kube-api-access-97dqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:12.524137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.524089    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:12.524137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.524108    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8120ce2e-a4af-4183-b215-cd1c31b31142-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:12.524137 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:12.524118    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97dqg\" (UniqueName: \"kubernetes.io/projected/8120ce2e-a4af-4183-b215-cd1c31b31142-kube-api-access-97dqg\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:13.239901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:13.239879    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"
Oct 30 19:54:13.239901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:13.239893    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w" event={"ID":"8120ce2e-a4af-4183-b215-cd1c31b31142","Type":"ContainerDied","Data":"9953cec0d5605016c0394031732222fd159a4944dc8f7b3c07c714604f26e4fb"}
Oct 30 19:54:13.240104 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:13.239930    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9953cec0d5605016c0394031732222fd159a4944dc8f7b3c07c714604f26e4fb"
Oct 30 19:54:13.636727 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:54:13.636694    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:54:20.373348 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:20.373309    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:54:23.480222 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:23.480193    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:54:28.279190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.279160    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"]
Oct 30 19:54:28.279605 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.279592    6777 cpu_manager.go:414] "containerMap: removing container" podUID="8120ce2e-a4af-4183-b215-cd1c31b31142" containerName="revoke-keys"
Oct 30 19:54:28.279657 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.279606    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8120ce2e-a4af-4183-b215-cd1c31b31142" containerName="revoke-keys"
Oct 30 19:54:28.279695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.279679    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="8120ce2e-a4af-4183-b215-cd1c31b31142" containerName="revoke-keys"
Oct 30 19:54:28.282749 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.282730    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.289492 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.289473    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a89c71-openshift-default-dockercfg-75dkk\""
Oct 30 19:54:28.300344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.300322    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"]
Oct 30 19:54:28.354010 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.353978    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-credential-socket\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.354150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.354035    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-envoy\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.354199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.354172    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-socket\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.354246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.354199    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-certs\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.354246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.354224    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-istio-token\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.354359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.354285    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7482c3-0dcc-4494-a936-c365201c21ef-istiod-ca-cert\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.354359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.354318    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7482c3-0dcc-4494-a936-c365201c21ef-istio-podinfo\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.354451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.354420    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghcr\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-kube-api-access-jghcr\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.354500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.354451    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-data\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.455989 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.455951    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-credential-socket\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456010    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-envoy\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456079    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-socket\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456100    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-certs\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456131    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-istio-token\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456156    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7482c3-0dcc-4494-a936-c365201c21ef-istiod-ca-cert\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456180    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7482c3-0dcc-4494-a936-c365201c21ef-istio-podinfo\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456212    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghcr\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-kube-api-access-jghcr\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456238    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-data\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456446    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-certs\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456447    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-credential-socket\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456536    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-socket\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.456771 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.456621    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-data\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.457170 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.457147    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7482c3-0dcc-4494-a936-c365201c21ef-istiod-ca-cert\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.458538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.458513    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-envoy\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.458826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.458805    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7482c3-0dcc-4494-a936-c365201c21ef-istio-podinfo\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.491313 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.491286    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-istio-token\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.491837 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.491816    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghcr\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-kube-api-access-jghcr\") pod \"e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") " pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.595632 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.595581    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:28.870429 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.870403    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"]
Oct 30 19:54:28.871584 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:54:28.871559    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7482c3_0dcc_4494_a936_c365201c21ef.slice/crio-8a4d6099fa77701a8ae64673ed6c88bb66e436ae28b5d8dc55164b95ea52cd1b WatchSource:0}: Error finding container 8a4d6099fa77701a8ae64673ed6c88bb66e436ae28b5d8dc55164b95ea52cd1b: Status 404 returned error can't find the container with id 8a4d6099fa77701a8ae64673ed6c88bb66e436ae28b5d8dc55164b95ea52cd1b
Oct 30 19:54:28.873525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.873495    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:28.873630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.873567    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:28.873630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:28.873612    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:29.328834 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:29.328801    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" event={"ID":"da7482c3-0dcc-4494-a936-c365201c21ef","Type":"ContainerStarted","Data":"5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385"}
Oct 30 19:54:29.329289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:29.328847    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" event={"ID":"da7482c3-0dcc-4494-a936-c365201c21ef","Type":"ContainerStarted","Data":"8a4d6099fa77701a8ae64673ed6c88bb66e436ae28b5d8dc55164b95ea52cd1b"}
Oct 30 19:54:29.353938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:29.353894    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" podStartSLOduration=1.353882131 podStartE2EDuration="1.353882131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:29.352697835 +0000 UTC m=+2851.349613218" watchObservedRunningTime="2026-10-30 19:54:29.353882131 +0000 UTC m=+2851.350797514"
Oct 30 19:54:29.595973 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:29.595904    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:29.600695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:29.600672    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:30.029745 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:30.029715    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"]
Oct 30 19:54:30.032248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:30.032222    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255d55c5-f5trd"]
Oct 30 19:54:30.111169 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:30.111144    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bbcbd7-69f3-402c-8206-2c050904b75d" path="/var/lib/kubelet/pods/d9bbcbd7-69f3-402c-8206-2c050904b75d/volumes"
Oct 30 19:54:30.334480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:30.334420    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:30.335463 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:30.335445    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:32.975884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:32.975855    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"]
Oct 30 19:54:32.980110 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:32.980090    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:32.989171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:32.989150    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e1653398-dockercfg-vxvcw\""
Oct 30 19:54:32.990630 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:32.990607    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"]
Oct 30 19:54:33.047235 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:33.047209    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a89c71\""
Oct 30 19:54:33.101367 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:33.101334    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c\" (UID: \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:33.101582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:33.101477    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwcsd\" (UniqueName: \"kubernetes.io/projected/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-kube-api-access-hwcsd\") pod \"payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c\" (UID: \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:33.202413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:33.202381    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwcsd\" (UniqueName: \"kubernetes.io/projected/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-kube-api-access-hwcsd\") pod \"payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c\" (UID: \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:33.202563 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:33.202450    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c\" (UID: \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:33.203136 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:33.203116    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c\" (UID: \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:33.214897 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:33.214867    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwcsd\" (UniqueName: \"kubernetes.io/projected/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-kube-api-access-hwcsd\") pod \"payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c\" (UID: \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:33.294730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:33.294698    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:33.571641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:33.571619    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"]
Oct 30 19:54:33.573818 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:54:33.573791    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50d9759_f8b0_4de5_8fe6_f9324af1dcea.slice/crio-62b6850c4b4398813d0b9fdbd98c75cb72df854db1cfb85a52c0671009935036 WatchSource:0}: Error finding container 62b6850c4b4398813d0b9fdbd98c75cb72df854db1cfb85a52c0671009935036: Status 404 returned error can't find the container with id 62b6850c4b4398813d0b9fdbd98c75cb72df854db1cfb85a52c0671009935036
Oct 30 19:54:34.359665 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:34.359627    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c" event={"ID":"b50d9759-f8b0-4de5-8fe6-f9324af1dcea","Type":"ContainerStarted","Data":"806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68"}
Oct 30 19:54:34.360047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:34.359687    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c" event={"ID":"b50d9759-f8b0-4de5-8fe6-f9324af1dcea","Type":"ContainerStarted","Data":"62b6850c4b4398813d0b9fdbd98c75cb72df854db1cfb85a52c0671009935036"}
Oct 30 19:54:34.360047 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:34.359738    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:34.380360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:34.380298    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c" podStartSLOduration=2.380286626 podStartE2EDuration="2.380286626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:34.378438858 +0000 UTC m=+2856.375354242" watchObservedRunningTime="2026-10-30 19:54:34.380286626 +0000 UTC m=+2856.377202009"
Oct 30 19:54:35.260978 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:35.260945    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:54:43.638619 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:54:43.638573    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:54:44.426229 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:44.426198    6777 generic.go:356] "Generic (PLEG): container finished" podID="da7482c3-0dcc-4494-a936-c365201c21ef" containerID="5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385" exitCode=0
Oct 30 19:54:44.426401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:44.426236    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" event={"ID":"da7482c3-0dcc-4494-a936-c365201c21ef","Type":"ContainerDied","Data":"5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385"}
Oct 30 19:54:44.426666 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:44.426648    6777 scope.go:122] "RemoveContainer" containerID="5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385"
Oct 30 19:54:44.427819 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:44.427793    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:44.427885 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:44.427863    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:44.427928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:44.427894    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:54:45.167084 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:45.155676    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:54:45.334928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:45.334894    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:45.377996 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:45.377963    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:54:45.434431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:45.434345    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" event={"ID":"da7482c3-0dcc-4494-a936-c365201c21ef","Type":"ContainerStarted","Data":"c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e"}
Oct 30 19:54:45.596246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:45.596211    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:45.597454 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:45.597427    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body=
Oct 30 19:54:45.597590 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:45.597476    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused"
Oct 30 19:54:46.596892 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:46.596863    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body=
Oct 30 19:54:46.597304 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:46.596935    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused"
Oct 30 19:54:47.596718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:47.596684    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body=
Oct 30 19:54:47.596901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:47.596737    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused"
Oct 30 19:54:48.596330 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:48.596291    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body=
Oct 30 19:54:48.596802 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:48.596345    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused"
Oct 30 19:54:49.596572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:49.596536    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body=
Oct 30 19:54:49.596963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:49.596592    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused"
Oct 30 19:54:50.715095 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:50.715046    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:50.715568 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:50.715544    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:54:50.716748 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:54:50.716720    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:55:00.661424 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:00.661385    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:55:03.983132 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:03.983089    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"]
Oct 30 19:55:03.987196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:03.987174    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:03.996245 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:03.996209    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:55:03.996381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:03.996267    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:03.996381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:03.996281    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:55:03.996381 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:03.996307    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:55:03.998122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:03.998100    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"]
Oct 30 19:55:04.056197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.056162    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71f2280-wmwxd\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.056370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.056283    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71f2280-wmwxd\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.056370 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.056312    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9v8\" (UniqueName: \"kubernetes.io/projected/24e36c66-49a4-4ce0-85f0-12ae0872c327-kube-api-access-js9v8\") pod \"maas-api-revoke-sub-e71f2280-wmwxd\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.157191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.157163    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71f2280-wmwxd\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.157382 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.157352    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71f2280-wmwxd\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.157520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.157417    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js9v8\" (UniqueName: \"kubernetes.io/projected/24e36c66-49a4-4ce0-85f0-12ae0872c327-kube-api-access-js9v8\") pod \"maas-api-revoke-sub-e71f2280-wmwxd\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.157863 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.157843    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71f2280-wmwxd\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.159600 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.159581    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71f2280-wmwxd\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.170575 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.170557    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9v8\" (UniqueName: \"kubernetes.io/projected/24e36c66-49a4-4ce0-85f0-12ae0872c327-kube-api-access-js9v8\") pod \"maas-api-revoke-sub-e71f2280-wmwxd\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.301149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.301131    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:04.572731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:04.572706    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"]
Oct 30 19:55:04.574467 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:55:04.574441    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e36c66_49a4_4ce0_85f0_12ae0872c327.slice/crio-442a757a78b650a35807dd3845da04552632d24fb239a6712ba09dc16e4d9c0f WatchSource:0}: Error finding container 442a757a78b650a35807dd3845da04552632d24fb239a6712ba09dc16e4d9c0f: Status 404 returned error can't find the container with id 442a757a78b650a35807dd3845da04552632d24fb239a6712ba09dc16e4d9c0f
Oct 30 19:55:05.567789 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:05.567755    6777 generic.go:356] "Generic (PLEG): container finished" podID="24e36c66-49a4-4ce0-85f0-12ae0872c327" containerID="0121b97ccead723b2195bb833828f3b4678b01ea23837871247b03d255c045f2" exitCode=0
Oct 30 19:55:05.568158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:05.567807    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd" event={"ID":"24e36c66-49a4-4ce0-85f0-12ae0872c327","Type":"ContainerDied","Data":"0121b97ccead723b2195bb833828f3b4678b01ea23837871247b03d255c045f2"}
Oct 30 19:55:05.568158 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:05.567862    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd" event={"ID":"24e36c66-49a4-4ce0-85f0-12ae0872c327","Type":"ContainerStarted","Data":"442a757a78b650a35807dd3845da04552632d24fb239a6712ba09dc16e4d9c0f"}
Oct 30 19:55:06.598099 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.595520    6777 scope.go:122] "RemoveContainer" containerID="f2aace53d95d8511ea3b3363f8460f04c8334684e0e7a88e4646b58b45c5d55a"
Oct 30 19:55:06.612312 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.612288    6777 scope.go:122] "RemoveContainer" containerID="8aefbd41eb22263aa92443582e9ca8a4e23babbef91f48845a78d934450369f9"
Oct 30 19:55:06.621935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.621911    6777 scope.go:122] "RemoveContainer" containerID="ec475bc82e1b5a809016b79181425e7d9f24af0890789f773d2b6610f08f86ec"
Oct 30 19:55:06.639886 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.635713    6777 scope.go:122] "RemoveContainer" containerID="8e5652556fbc732c5be138ab839820b58851415b106178a326e52adacba25987"
Oct 30 19:55:06.647866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.647841    6777 scope.go:122] "RemoveContainer" containerID="884decb53796c3f4c0a50218e289e2ce5988860ea2a1d379e35c256064d9009d"
Oct 30 19:55:06.659144 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.659121    6777 scope.go:122] "RemoveContainer" containerID="86aa1ff670fe428b9bf2f5d97e362c11a2715bfacdf1f3c5ddb83d04ca1094f6"
Oct 30 19:55:06.736002 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.735981    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:06.782121 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.782095    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-xks-ca\") pod \"24e36c66-49a4-4ce0-85f0-12ae0872c327\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") "
Oct 30 19:55:06.782234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.782190    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e36c66-49a4-4ce0-85f0-12ae0872c327-kube-api-access-js9v8\" (UniqueName: \"kubernetes.io/projected/24e36c66-49a4-4ce0-85f0-12ae0872c327-kube-api-access-js9v8\") pod \"24e36c66-49a4-4ce0-85f0-12ae0872c327\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") "
Oct 30 19:55:06.782280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.782246    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-openshift-ca\") pod \"24e36c66-49a4-4ce0-85f0-12ae0872c327\" (UID: \"24e36c66-49a4-4ce0-85f0-12ae0872c327\") "
Oct 30 19:55:06.782615 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.782593    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-openshift-ca" pod "24e36c66-49a4-4ce0-85f0-12ae0872c327" (UID: "24e36c66-49a4-4ce0-85f0-12ae0872c327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:06.782681 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.782665    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:06.784324 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.784302    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-xks-ca" pod "24e36c66-49a4-4ce0-85f0-12ae0872c327" (UID: "24e36c66-49a4-4ce0-85f0-12ae0872c327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:06.784431 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.784300    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e36c66-49a4-4ce0-85f0-12ae0872c327-kube-api-access-js9v8" pod "24e36c66-49a4-4ce0-85f0-12ae0872c327" (UID: "24e36c66-49a4-4ce0-85f0-12ae0872c327"). InnerVolumeSpecName "kube-api-access-js9v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:06.883641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.883586    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e36c66-49a4-4ce0-85f0-12ae0872c327-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:06.883641 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:06.883610    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js9v8\" (UniqueName: \"kubernetes.io/projected/24e36c66-49a4-4ce0-85f0-12ae0872c327-kube-api-access-js9v8\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:07.580845 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:07.580816    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"
Oct 30 19:55:07.581027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:07.580843    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd" event={"ID":"24e36c66-49a4-4ce0-85f0-12ae0872c327","Type":"ContainerDied","Data":"442a757a78b650a35807dd3845da04552632d24fb239a6712ba09dc16e4d9c0f"}
Oct 30 19:55:07.581027 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:07.580890    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442a757a78b650a35807dd3845da04552632d24fb239a6712ba09dc16e4d9c0f"
Oct 30 19:55:13.637766 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:55:13.637693    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:55:14.971291 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:14.971260    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:55:15.066122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.066095    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"]
Oct 30 19:55:15.066553 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.066540    6777 cpu_manager.go:414] "containerMap: removing container" podUID="24e36c66-49a4-4ce0-85f0-12ae0872c327" containerName="revoke-keys"
Oct 30 19:55:15.066599 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.066555    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e36c66-49a4-4ce0-85f0-12ae0872c327" containerName="revoke-keys"
Oct 30 19:55:15.066635 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.066629    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e36c66-49a4-4ce0-85f0-12ae0872c327" containerName="revoke-keys"
Oct 30 19:55:15.070032 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.070014    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.089921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.089899    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:55:15.090108 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.090090    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:15.090194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.090139    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:55:15.090194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.090153    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:55:15.090505 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.090487    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"]
Oct 30 19:55:15.161602 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.161577    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4g6\" (UniqueName: \"kubernetes.io/projected/1ec1c864-b459-4802-9129-b6c9cb6a7b19-kube-api-access-8h4g6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.161752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.161636    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.161752 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.161730    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.263143 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.263084    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4g6\" (UniqueName: \"kubernetes.io/projected/1ec1c864-b459-4802-9129-b6c9cb6a7b19-kube-api-access-8h4g6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.263248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.263156    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.263248 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.263190    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.263746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.263729    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.265464 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.265448    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.277937 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.277919    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4g6\" (UniqueName: \"kubernetes.io/projected/1ec1c864-b459-4802-9129-b6c9cb6a7b19-kube-api-access-8h4g6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.382475 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.382438    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:15.655376 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:15.655349    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"]
Oct 30 19:55:15.656644 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:55:15.656619    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec1c864_b459_4802_9129_b6c9cb6a7b19.slice/crio-823e4e005afeb97bb86b4382d7565f4f376bea10fdf0d308751837452f9cc614 WatchSource:0}: Error finding container 823e4e005afeb97bb86b4382d7565f4f376bea10fdf0d308751837452f9cc614: Status 404 returned error can't find the container with id 823e4e005afeb97bb86b4382d7565f4f376bea10fdf0d308751837452f9cc614
Oct 30 19:55:16.639448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:16.639416    6777 generic.go:356] "Generic (PLEG): container finished" podID="1ec1c864-b459-4802-9129-b6c9cb6a7b19" containerID="915f59d9266340a0b8c6ee423e8d49fa9d9c81d21140e1b73bf6d8d92f9dedf9" exitCode=0
Oct 30 19:55:16.639894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:16.639503    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp" event={"ID":"1ec1c864-b459-4802-9129-b6c9cb6a7b19","Type":"ContainerDied","Data":"915f59d9266340a0b8c6ee423e8d49fa9d9c81d21140e1b73bf6d8d92f9dedf9"}
Oct 30 19:55:16.639894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:16.639555    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp" event={"ID":"1ec1c864-b459-4802-9129-b6c9cb6a7b19","Type":"ContainerStarted","Data":"823e4e005afeb97bb86b4382d7565f4f376bea10fdf0d308751837452f9cc614"}
Oct 30 19:55:17.772985 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.772959    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:17.888379 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.888350    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-xks-ca\") pod \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") "
Oct 30 19:55:17.888556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.888420    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec1c864-b459-4802-9129-b6c9cb6a7b19-kube-api-access-8h4g6\" (UniqueName: \"kubernetes.io/projected/1ec1c864-b459-4802-9129-b6c9cb6a7b19-kube-api-access-8h4g6\") pod \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") "
Oct 30 19:55:17.888556 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.888473    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-openshift-ca\") pod \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\" (UID: \"1ec1c864-b459-4802-9129-b6c9cb6a7b19\") "
Oct 30 19:55:17.888889 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.888855    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-openshift-ca" pod "1ec1c864-b459-4802-9129-b6c9cb6a7b19" (UID: "1ec1c864-b459-4802-9129-b6c9cb6a7b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:17.890451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.890425    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec1c864-b459-4802-9129-b6c9cb6a7b19-kube-api-access-8h4g6" pod "1ec1c864-b459-4802-9129-b6c9cb6a7b19" (UID: "1ec1c864-b459-4802-9129-b6c9cb6a7b19"). InnerVolumeSpecName "kube-api-access-8h4g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:17.890584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.890455    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-xks-ca" pod "1ec1c864-b459-4802-9129-b6c9cb6a7b19" (UID: "1ec1c864-b459-4802-9129-b6c9cb6a7b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:17.989893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.989855    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h4g6\" (UniqueName: \"kubernetes.io/projected/1ec1c864-b459-4802-9129-b6c9cb6a7b19-kube-api-access-8h4g6\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:17.989893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.989893    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:17.990127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:17.989907    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec1c864-b459-4802-9129-b6c9cb6a7b19-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:18.026070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:18.026028    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"]
Oct 30 19:55:18.029171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:18.029148    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lgtkg"]
Oct 30 19:55:18.110631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:18.110607    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179707e6-345c-4b9d-a10b-2fa007ae5bb0" path="/var/lib/kubelet/pods/179707e6-345c-4b9d-a10b-2fa007ae5bb0/volumes"
Oct 30 19:55:18.654864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:18.654833    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"
Oct 30 19:55:18.654864 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:18.654849    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp" event={"ID":"1ec1c864-b459-4802-9129-b6c9cb6a7b19","Type":"ContainerDied","Data":"823e4e005afeb97bb86b4382d7565f4f376bea10fdf0d308751837452f9cc614"}
Oct 30 19:55:18.655126 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:18.654914    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="823e4e005afeb97bb86b4382d7565f4f376bea10fdf0d308751837452f9cc614"
Oct 30 19:55:25.186078 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:25.186024    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"]
Oct 30 19:55:25.616396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:25.616364    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"]
Oct 30 19:55:25.619508 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:25.619483    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a89c71-8bsfp"]
Oct 30 19:55:26.110747 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:26.110715    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec1c864-b459-4802-9129-b6c9cb6a7b19" path="/var/lib/kubelet/pods/1ec1c864-b459-4802-9129-b6c9cb6a7b19/volumes"
Oct 30 19:55:27.158315 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:27.158285    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:55:30.186435 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:30.186402    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c" podUID="b50d9759-f8b0-4de5-8fe6-f9324af1dcea" containerName="payload-pre-processing" containerID="cri-o://806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68" gracePeriod=25
Oct 30 19:55:33.018168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:33.018130    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"]
Oct 30 19:55:33.019070 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:33.019004    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy" containerID="cri-o://c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e" gracePeriod=30
Oct 30 19:55:34.272565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.272527    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"]
Oct 30 19:55:34.277788 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.277762    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hz89w"]
Oct 30 19:55:34.298899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.298868    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"]
Oct 30 19:55:34.299378 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.299360    6777 cpu_manager.go:414] "containerMap: removing container" podUID="1ec1c864-b459-4802-9129-b6c9cb6a7b19" containerName="revoke-keys"
Oct 30 19:55:34.299476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.299379    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec1c864-b459-4802-9129-b6c9cb6a7b19" containerName="revoke-keys"
Oct 30 19:55:34.299476 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.299473    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec1c864-b459-4802-9129-b6c9cb6a7b19" containerName="revoke-keys"
Oct 30 19:55:34.302765 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.302746    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.310565 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.310544    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:34.310772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.310587    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:55:34.310772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.310639    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:55:34.310772 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.310651    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:55:34.311404 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.311386    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"]
Oct 30 19:55:34.437451 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.437424    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7dkr\" (UniqueName: \"kubernetes.io/projected/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-kube-api-access-s7dkr\") pod \"maas-api-revoke-sub-3d4156f1-vd2bc\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.437564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.437462    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vd2bc\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.437564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.437492    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vd2bc\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.538263 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.538240    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7dkr\" (UniqueName: \"kubernetes.io/projected/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-kube-api-access-s7dkr\") pod \"maas-api-revoke-sub-3d4156f1-vd2bc\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.538372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.538280    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vd2bc\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.538372 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.538317    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vd2bc\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.538868 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.538849    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vd2bc\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.540649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.540628    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vd2bc\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.551701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.551681    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7dkr\" (UniqueName: \"kubernetes.io/projected/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-kube-api-access-s7dkr\") pod \"maas-api-revoke-sub-3d4156f1-vd2bc\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.617344 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.617324    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:34.893917 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:34.893891    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"]
Oct 30 19:55:34.895670 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:55:34.895638    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cc9725_177a_4ba3_9dbb_ca262b3e8ef8.slice/crio-a0604b176c7dc3f75380712462a0399258b443585b2c3e0b10711cc11669f232 WatchSource:0}: Error finding container a0604b176c7dc3f75380712462a0399258b443585b2c3e0b10711cc11669f232: Status 404 returned error can't find the container with id a0604b176c7dc3f75380712462a0399258b443585b2c3e0b10711cc11669f232
Oct 30 19:55:35.366396 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:35.366361    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c" podUID="b50d9759-f8b0-4de5-8fe6-f9324af1dcea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:55:35.718160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:35.718131    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 19:55:35.718284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:35.718181    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:55:35.748052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:35.748029    6777 generic.go:356] "Generic (PLEG): container finished" podID="93cc9725-177a-4ba3-9dbb-ca262b3e8ef8" containerID="60c08de166ee5266d3a8374581f0148478e00899becd758887e27b8fed9d1811" exitCode=0
Oct 30 19:55:35.748175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:35.748102    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc" event={"ID":"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8","Type":"ContainerDied","Data":"60c08de166ee5266d3a8374581f0148478e00899becd758887e27b8fed9d1811"}
Oct 30 19:55:35.748175 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:35.748147    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc" event={"ID":"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8","Type":"ContainerStarted","Data":"a0604b176c7dc3f75380712462a0399258b443585b2c3e0b10711cc11669f232"}
Oct 30 19:55:36.110935 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:36.110901    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8120ce2e-a4af-4183-b215-cd1c31b31142" path="/var/lib/kubelet/pods/8120ce2e-a4af-4183-b215-cd1c31b31142/volumes"
Oct 30 19:55:36.887489 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:36.887467    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:36.962360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:36.962332    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-kube-api-access-s7dkr\" (UniqueName: \"kubernetes.io/projected/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-kube-api-access-s7dkr\") pod \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") "
Oct 30 19:55:36.962511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:36.962396    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-openshift-ca\") pod \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") "
Oct 30 19:55:36.962511 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:36.962455    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-xks-ca\") pod \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\" (UID: \"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8\") "
Oct 30 19:55:36.962826 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:36.962801    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-openshift-ca" pod "93cc9725-177a-4ba3-9dbb-ca262b3e8ef8" (UID: "93cc9725-177a-4ba3-9dbb-ca262b3e8ef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:36.962930 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:36.962914    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:36.964538 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:36.964512    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-kube-api-access-s7dkr" pod "93cc9725-177a-4ba3-9dbb-ca262b3e8ef8" (UID: "93cc9725-177a-4ba3-9dbb-ca262b3e8ef8"). InnerVolumeSpecName "kube-api-access-s7dkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:36.964631 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:36.964614    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-xks-ca" pod "93cc9725-177a-4ba3-9dbb-ca262b3e8ef8" (UID: "93cc9725-177a-4ba3-9dbb-ca262b3e8ef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:37.063801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:37.063771    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7dkr\" (UniqueName: \"kubernetes.io/projected/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-kube-api-access-s7dkr\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:37.063801 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:37.063799    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:37.760383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:37.760356    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"
Oct 30 19:55:37.760383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:37.760368    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc" event={"ID":"93cc9725-177a-4ba3-9dbb-ca262b3e8ef8","Type":"ContainerDied","Data":"a0604b176c7dc3f75380712462a0399258b443585b2c3e0b10711cc11669f232"}
Oct 30 19:55:37.760581 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:37.760432    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0604b176c7dc3f75380712462a0399258b443585b2c3e0b10711cc11669f232"
Oct 30 19:55:38.273004 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.272979    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:55:38.374582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374511    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-data\") pod \"da7482c3-0dcc-4494-a936-c365201c21ef\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") "
Oct 30 19:55:38.374582 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374567    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-certs\") pod \"da7482c3-0dcc-4494-a936-c365201c21ef\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") "
Oct 30 19:55:38.374780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374609    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7482c3-0dcc-4494-a936-c365201c21ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7482c3-0dcc-4494-a936-c365201c21ef-istiod-ca-cert\") pod \"da7482c3-0dcc-4494-a936-c365201c21ef\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") "
Oct 30 19:55:38.374780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374644    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-credential-socket\") pod \"da7482c3-0dcc-4494-a936-c365201c21ef\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") "
Oct 30 19:55:38.374780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374677    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-envoy\") pod \"da7482c3-0dcc-4494-a936-c365201c21ef\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") "
Oct 30 19:55:38.374780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374711    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da7482c3-0dcc-4494-a936-c365201c21ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7482c3-0dcc-4494-a936-c365201c21ef-istio-podinfo\") pod \"da7482c3-0dcc-4494-a936-c365201c21ef\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") "
Oct 30 19:55:38.374780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374760    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-certs" pod "da7482c3-0dcc-4494-a936-c365201c21ef" (UID: "da7482c3-0dcc-4494-a936-c365201c21ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:38.374780 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374771    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-kube-api-access-jghcr\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-kube-api-access-jghcr\") pod \"da7482c3-0dcc-4494-a936-c365201c21ef\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") "
Oct 30 19:55:38.375130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374793    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-data" pod "da7482c3-0dcc-4494-a936-c365201c21ef" (UID: "da7482c3-0dcc-4494-a936-c365201c21ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:38.375130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374893    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-istio-token\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-istio-token\") pod \"da7482c3-0dcc-4494-a936-c365201c21ef\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") "
Oct 30 19:55:38.375130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374927    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-socket\") pod \"da7482c3-0dcc-4494-a936-c365201c21ef\" (UID: \"da7482c3-0dcc-4494-a936-c365201c21ef\") "
Oct 30 19:55:38.375130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.374957    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-credential-socket" pod "da7482c3-0dcc-4494-a936-c365201c21ef" (UID: "da7482c3-0dcc-4494-a936-c365201c21ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:38.375130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.375085    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7482c3-0dcc-4494-a936-c365201c21ef-istiod-ca-cert" pod "da7482c3-0dcc-4494-a936-c365201c21ef" (UID: "da7482c3-0dcc-4494-a936-c365201c21ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:38.375393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.375314    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.375393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.375336    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.375393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.375340    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-socket" pod "da7482c3-0dcc-4494-a936-c365201c21ef" (UID: "da7482c3-0dcc-4494-a936-c365201c21ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:38.375393 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.375350    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7482c3-0dcc-4494-a936-c365201c21ef-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.375525 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.375391    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.376878 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.376857    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da7482c3-0dcc-4494-a936-c365201c21ef-istio-podinfo" pod "da7482c3-0dcc-4494-a936-c365201c21ef" (UID: "da7482c3-0dcc-4494-a936-c365201c21ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:38.376991 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.376977    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-kube-api-access-jghcr" pod "da7482c3-0dcc-4494-a936-c365201c21ef" (UID: "da7482c3-0dcc-4494-a936-c365201c21ef"). InnerVolumeSpecName "kube-api-access-jghcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:38.377219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.377205    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-envoy" pod "da7482c3-0dcc-4494-a936-c365201c21ef" (UID: "da7482c3-0dcc-4494-a936-c365201c21ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:38.377319 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.377297    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-istio-token" pod "da7482c3-0dcc-4494-a936-c365201c21ef" (UID: "da7482c3-0dcc-4494-a936-c365201c21ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:38.476157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.476135    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.476157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.476154    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7482c3-0dcc-4494-a936-c365201c21ef-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.476284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.476163    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jghcr\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-kube-api-access-jghcr\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.476284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.476174    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7482c3-0dcc-4494-a936-c365201c21ef-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.476284 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.476183    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7482c3-0dcc-4494-a936-c365201c21ef-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.767311 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.767260    6777 generic.go:356] "Generic (PLEG): container finished" podID="da7482c3-0dcc-4494-a936-c365201c21ef" containerID="c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e" exitCode=0
Oct 30 19:55:38.767406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.767307    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"
Oct 30 19:55:38.767406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.767360    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" event={"ID":"da7482c3-0dcc-4494-a936-c365201c21ef","Type":"ContainerDied","Data":"c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e"}
Oct 30 19:55:38.767480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.767407    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c" event={"ID":"da7482c3-0dcc-4494-a936-c365201c21ef","Type":"ContainerDied","Data":"8a4d6099fa77701a8ae64673ed6c88bb66e436ae28b5d8dc55164b95ea52cd1b"}
Oct 30 19:55:38.767480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.767431    6777 scope.go:122] "RemoveContainer" containerID="c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e"
Oct 30 19:55:38.776758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.776742    6777 scope.go:122] "RemoveContainer" containerID="5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385"
Oct 30 19:55:38.784510 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.784494    6777 scope.go:122] "RemoveContainer" containerID="c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e"
Oct 30 19:55:38.784759 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:55:38.784743    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e\": container with ID starting with c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e not found: ID does not exist" containerID="c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e"
Oct 30 19:55:38.784811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.784767    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e"} err="failed to get container status \"c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e\": rpc error: code = NotFound desc = could not find container \"c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e\": container with ID starting with c508607ac97784d55fa14a4334a060aeaa05d62c555fb614c6a1e2977dd0ff6e not found: ID does not exist"
Oct 30 19:55:38.784811 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.784781    6777 scope.go:122] "RemoveContainer" containerID="5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385"
Oct 30 19:55:38.784989 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:55:38.784974    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385\": container with ID starting with 5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385 not found: ID does not exist" containerID="5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385"
Oct 30 19:55:38.785035 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.784995    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385"} err="failed to get container status \"5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385\": rpc error: code = NotFound desc = could not find container \"5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385\": container with ID starting with 5c2e3f0860cbb2e4ab62cc2027917bd9bf84c64053e886b73a1cc0a61be4c385 not found: ID does not exist"
Oct 30 19:55:38.790741 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.790723    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"]
Oct 30 19:55:38.795808 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:38.795787    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a89c71-openshift-default-7f5ff58fb9d2r5c"]
Oct 30 19:55:40.110965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:40.110928    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" path="/var/lib/kubelet/pods/da7482c3-0dcc-4494-a936-c365201c21ef/volumes"
Oct 30 19:55:43.637221 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:55:43.637189    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:55:45.366695 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:45.366658    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c" podUID="b50d9759-f8b0-4de5-8fe6-f9324af1dcea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:55:45.564877 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:45.564846    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:55:47.432944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:47.432910    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:47.433611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:47.433571    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:49.790873 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:49.790835    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:55:50.445595 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.445537    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:55:50.521151 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.521123    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-plugins-config-volume\") pod \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\" (UID: \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\") "
Oct 30 19:55:50.521288 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.521239    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-kube-api-access-hwcsd\" (UniqueName: \"kubernetes.io/projected/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-kube-api-access-hwcsd\") pod \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\" (UID: \"b50d9759-f8b0-4de5-8fe6-f9324af1dcea\") "
Oct 30 19:55:50.521467 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.521446    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-plugins-config-volume" pod "b50d9759-f8b0-4de5-8fe6-f9324af1dcea" (UID: "b50d9759-f8b0-4de5-8fe6-f9324af1dcea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:50.523224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.523202    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-kube-api-access-hwcsd" pod "b50d9759-f8b0-4de5-8fe6-f9324af1dcea" (UID: "b50d9759-f8b0-4de5-8fe6-f9324af1dcea"). InnerVolumeSpecName "kube-api-access-hwcsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:50.622662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.622630    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:50.622662 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.622661    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwcsd\" (UniqueName: \"kubernetes.io/projected/b50d9759-f8b0-4de5-8fe6-f9324af1dcea-kube-api-access-hwcsd\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:50.863484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.863460    6777 generic.go:356] "Generic (PLEG): container finished" podID="b50d9759-f8b0-4de5-8fe6-f9324af1dcea" containerID="806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68" exitCode=0
Oct 30 19:55:50.863854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.863501    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"
Oct 30 19:55:50.863854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.863549    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c" event={"ID":"b50d9759-f8b0-4de5-8fe6-f9324af1dcea","Type":"ContainerDied","Data":"806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68"}
Oct 30 19:55:50.863854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.863646    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c" event={"ID":"b50d9759-f8b0-4de5-8fe6-f9324af1dcea","Type":"ContainerDied","Data":"62b6850c4b4398813d0b9fdbd98c75cb72df854db1cfb85a52c0671009935036"}
Oct 30 19:55:50.863854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.863679    6777 scope.go:122] "RemoveContainer" containerID="806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68"
Oct 30 19:55:50.875150 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.875134    6777 scope.go:122] "RemoveContainer" containerID="806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68"
Oct 30 19:55:50.875432 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:55:50.875413    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68\": container with ID starting with 806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68 not found: ID does not exist" containerID="806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68"
Oct 30 19:55:50.875494 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.875441    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68"} err="failed to get container status \"806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68\": rpc error: code = NotFound desc = could not find container \"806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68\": container with ID starting with 806142d728a0c30356719bbe49fda168da59c899717d3db7b08b864c836baa68 not found: ID does not exist"
Oct 30 19:55:50.889824 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.889801    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"]
Oct 30 19:55:50.897069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:50.897037    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a89c71-6d898kpd7c"]
Oct 30 19:55:52.111709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:55:52.111677    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50d9759-f8b0-4de5-8fe6-f9324af1dcea" path="/var/lib/kubelet/pods/b50d9759-f8b0-4de5-8fe6-f9324af1dcea/volumes"
Oct 30 19:56:02.029497 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:02.029465    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"]
Oct 30 19:56:02.032506 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:02.032480    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-457fg"]
Oct 30 19:56:02.111129 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:02.111096    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0c4eed-9b82-4e95-af23-085885d63338" path="/var/lib/kubelet/pods/ed0c4eed-9b82-4e95-af23-085885d63338/volumes"
Oct 30 19:56:06.711700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:06.711668    6777 scope.go:122] "RemoveContainer" containerID="decbc999e7e0dc4cbe789f27ef4e3d6637d4fb24ce2d7e405f59769ca1335c61"
Oct 30 19:56:13.637347 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:56:13.637314    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:56:20.026997 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:20.026950    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"]
Oct 30 19:56:20.029842 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:20.029808    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jv7h9"]
Oct 30 19:56:20.110952 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:20.110926    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b47a90c-88f3-4c12-873c-f5dd4cc5ab63" path="/var/lib/kubelet/pods/2b47a90c-88f3-4c12-873c-f5dd4cc5ab63/volumes"
Oct 30 19:56:32.036190 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:32.036151    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"]
Oct 30 19:56:32.039419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:32.039391    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mjgzk"]
Oct 30 19:56:32.111016 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:32.110986    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e4517f-5386-4fb9-8c67-4026e4fe81b5" path="/var/lib/kubelet/pods/64e4517f-5386-4fb9-8c67-4026e4fe81b5/volumes"
Oct 30 19:56:43.638012 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:56:43.637973    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:56:58.033117 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:58.033082    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"]
Oct 30 19:56:58.039756 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:58.039730    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5hw26"]
Oct 30 19:56:58.111023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:56:58.110976    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c227135d-ab77-463d-936e-a574d9c3ac2c" path="/var/lib/kubelet/pods/c227135d-ab77-463d-936e-a574d9c3ac2c/volumes"
Oct 30 19:57:06.740576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:06.740545    6777 scope.go:122] "RemoveContainer" containerID="c1a4bf6ea589d219bb4e7831aea284ac8a64aa7bc7697fd73973c539fa90a0e1"
Oct 30 19:57:06.748843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:06.748824    6777 scope.go:122] "RemoveContainer" containerID="d9606f14f6021ea060aa3deca1eee78f26ddfdf6c129c2c8d83c45845b621266"
Oct 30 19:57:09.027496 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:09.027467    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"]
Oct 30 19:57:09.030675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:09.030652    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t7z6w"]
Oct 30 19:57:10.110271 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:10.110242    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eccf5bbe-61df-4855-b55c-54af345ef794" path="/var/lib/kubelet/pods/eccf5bbe-61df-4855-b55c-54af345ef794/volumes"
Oct 30 19:57:13.636538 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:57:13.636500    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:57:13.851480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:13.851451    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:57:17.874452 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874419    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"]
Oct 30 19:57:17.874854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874841    6777 cpu_manager.go:414] "containerMap: removing container" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy"
Oct 30 19:57:17.874899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874854    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy"
Oct 30 19:57:17.874899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874882    6777 cpu_manager.go:414] "containerMap: removing container" podUID="93cc9725-177a-4ba3-9dbb-ca262b3e8ef8" containerName="revoke-keys"
Oct 30 19:57:17.874899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874887    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cc9725-177a-4ba3-9dbb-ca262b3e8ef8" containerName="revoke-keys"
Oct 30 19:57:17.875008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874907    6777 cpu_manager.go:414] "containerMap: removing container" podUID="b50d9759-f8b0-4de5-8fe6-f9324af1dcea" containerName="payload-pre-processing"
Oct 30 19:57:17.875008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874914    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50d9759-f8b0-4de5-8fe6-f9324af1dcea" containerName="payload-pre-processing"
Oct 30 19:57:17.875008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874927    6777 cpu_manager.go:414] "containerMap: removing container" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy"
Oct 30 19:57:17.875008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874932    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy"
Oct 30 19:57:17.875008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.874997    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy"
Oct 30 19:57:17.875008 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.875009    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cc9725-177a-4ba3-9dbb-ca262b3e8ef8" containerName="revoke-keys"
Oct 30 19:57:17.875218 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.875016    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50d9759-f8b0-4de5-8fe6-f9324af1dcea" containerName="payload-pre-processing"
Oct 30 19:57:17.879619 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.879597    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:17.889278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.889257    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"]
Oct 30 19:57:17.891210 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.891190    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:17.891316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.891241    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:57:17.891394 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.891382    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:57:17.891448 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.891388    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:57:17.972181 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.972155    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-86smx\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:17.972294 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.972209    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwmcb\" (UniqueName: \"kubernetes.io/projected/66012094-5a47-4234-8c38-13dcd59901ec-kube-api-access-gwmcb\") pod \"maas-api-revoke-sub-d576e68b-86smx\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:17.972339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:17.972307    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-86smx\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:18.073383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:18.073358    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-86smx\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:18.073500 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:18.073456    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-86smx\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:18.073567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:18.073501    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwmcb\" (UniqueName: \"kubernetes.io/projected/66012094-5a47-4234-8c38-13dcd59901ec-kube-api-access-gwmcb\") pod \"maas-api-revoke-sub-d576e68b-86smx\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:18.074009 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:18.073988    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-86smx\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:18.075571 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:18.075550    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-86smx\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:18.087023 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:18.087005    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwmcb\" (UniqueName: \"kubernetes.io/projected/66012094-5a47-4234-8c38-13dcd59901ec-kube-api-access-gwmcb\") pod \"maas-api-revoke-sub-d576e68b-86smx\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:18.194037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:18.193990    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:18.467067 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:18.467036    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"]
Oct 30 19:57:18.469431 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:57:18.469406    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66012094_5a47_4234_8c38_13dcd59901ec.slice/crio-5e1af8458c1d7b74c3d9a567428f14d7d3f4da48f5d24f74a9a000c9d93f3cac WatchSource:0}: Error finding container 5e1af8458c1d7b74c3d9a567428f14d7d3f4da48f5d24f74a9a000c9d93f3cac: Status 404 returned error can't find the container with id 5e1af8458c1d7b74c3d9a567428f14d7d3f4da48f5d24f74a9a000c9d93f3cac
Oct 30 19:57:19.354317 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:19.354291    6777 generic.go:356] "Generic (PLEG): container finished" podID="66012094-5a47-4234-8c38-13dcd59901ec" containerID="f5f0d80035699c86c25376452f6ed6908fef8a1b5acf8d854daed46bad26e309" exitCode=0
Oct 30 19:57:19.354620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:19.354371    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx" event={"ID":"66012094-5a47-4234-8c38-13dcd59901ec","Type":"ContainerDied","Data":"f5f0d80035699c86c25376452f6ed6908fef8a1b5acf8d854daed46bad26e309"}
Oct 30 19:57:19.354620 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:19.354419    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx" event={"ID":"66012094-5a47-4234-8c38-13dcd59901ec","Type":"ContainerStarted","Data":"5e1af8458c1d7b74c3d9a567428f14d7d3f4da48f5d24f74a9a000c9d93f3cac"}
Oct 30 19:57:20.485527 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.485506    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:20.594413 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.594390    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-xks-ca\") pod \"66012094-5a47-4234-8c38-13dcd59901ec\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") "
Oct 30 19:57:20.594516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.594434    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66012094-5a47-4234-8c38-13dcd59901ec-kube-api-access-gwmcb\" (UniqueName: \"kubernetes.io/projected/66012094-5a47-4234-8c38-13dcd59901ec-kube-api-access-gwmcb\") pod \"66012094-5a47-4234-8c38-13dcd59901ec\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") "
Oct 30 19:57:20.594516 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.594481    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-openshift-ca\") pod \"66012094-5a47-4234-8c38-13dcd59901ec\" (UID: \"66012094-5a47-4234-8c38-13dcd59901ec\") "
Oct 30 19:57:20.594944 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.594917    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-openshift-ca" pod "66012094-5a47-4234-8c38-13dcd59901ec" (UID: "66012094-5a47-4234-8c38-13dcd59901ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:20.596483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.596455    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-xks-ca" pod "66012094-5a47-4234-8c38-13dcd59901ec" (UID: "66012094-5a47-4234-8c38-13dcd59901ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:20.596576 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.596553    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66012094-5a47-4234-8c38-13dcd59901ec-kube-api-access-gwmcb" pod "66012094-5a47-4234-8c38-13dcd59901ec" (UID: "66012094-5a47-4234-8c38-13dcd59901ec"). InnerVolumeSpecName "kube-api-access-gwmcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:20.696197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.696139    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwmcb\" (UniqueName: \"kubernetes.io/projected/66012094-5a47-4234-8c38-13dcd59901ec-kube-api-access-gwmcb\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:20.696197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.696161    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:20.696197 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:20.696175    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66012094-5a47-4234-8c38-13dcd59901ec-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:21.365647 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:21.365623    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"
Oct 30 19:57:21.365781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:21.365645    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx" event={"ID":"66012094-5a47-4234-8c38-13dcd59901ec","Type":"ContainerDied","Data":"5e1af8458c1d7b74c3d9a567428f14d7d3f4da48f5d24f74a9a000c9d93f3cac"}
Oct 30 19:57:21.365781 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:21.365679    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e1af8458c1d7b74c3d9a567428f14d7d3f4da48f5d24f74a9a000c9d93f3cac"
Oct 30 19:57:28.548336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:28.548305    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:57:29.026103 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:29.026002    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"]
Oct 30 19:57:29.029836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:29.029810    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p2v5b"]
Oct 30 19:57:30.110112 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:30.110079    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe40a99-8179-4deb-b3f5-299f4fafdd62" path="/var/lib/kubelet/pods/bfe40a99-8179-4deb-b3f5-299f4fafdd62/volumes"
Oct 30 19:57:32.427224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:32.427177    6777 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pthrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:57:32.427856 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:32.427808    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pthrg" podUID="8084f1f2-0fb7-4dbb-b255-4ffcef554e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:33.719982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.719951    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"]
Oct 30 19:57:33.720402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.720382    6777 cpu_manager.go:414] "containerMap: removing container" podUID="66012094-5a47-4234-8c38-13dcd59901ec" containerName="revoke-keys"
Oct 30 19:57:33.720402 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.720395    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66012094-5a47-4234-8c38-13dcd59901ec" containerName="revoke-keys"
Oct 30 19:57:33.720486 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.720480    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7482c3-0dcc-4494-a936-c365201c21ef" containerName="istio-proxy"
Oct 30 19:57:33.720522 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.720489    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="66012094-5a47-4234-8c38-13dcd59901ec" containerName="revoke-keys"
Oct 30 19:57:33.723638 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.723620    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:33.733042 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.733022    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:57:33.733189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.733175    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:57:33.733604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.733240    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:57:33.733604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.733249    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:33.734333 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.734288    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"]
Oct 30 19:57:33.737401 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.737377    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lbrqn\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:33.737498 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.737423    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lbrqn\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:33.737567 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.737546    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ltf\" (UniqueName: \"kubernetes.io/projected/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-kube-api-access-t4ltf\") pod \"maas-api-revoke-sub-d2bc93bc-lbrqn\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:33.838045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.838023    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lbrqn\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:33.838157 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.838103    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ltf\" (UniqueName: \"kubernetes.io/projected/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-kube-api-access-t4ltf\") pod \"maas-api-revoke-sub-d2bc93bc-lbrqn\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:33.838213 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.838182    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lbrqn\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:33.838751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.838731    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lbrqn\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:33.840238 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.840220    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lbrqn\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:33.850844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:33.850822    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ltf\" (UniqueName: \"kubernetes.io/projected/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-kube-api-access-t4ltf\") pod \"maas-api-revoke-sub-d2bc93bc-lbrqn\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:34.036713 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:34.036694    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:34.306338 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:34.306314    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"]
Oct 30 19:57:34.308289 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:57:34.308265    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abbb2ee_29fd_4f1c_af8b_cd2b72cf7c4a.slice/crio-3ca8e9d776e2b14a79b73df1324530045e9cfd59217562775db4c97a67485d07 WatchSource:0}: Error finding container 3ca8e9d776e2b14a79b73df1324530045e9cfd59217562775db4c97a67485d07: Status 404 returned error can't find the container with id 3ca8e9d776e2b14a79b73df1324530045e9cfd59217562775db4c97a67485d07
Oct 30 19:57:34.443495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:34.443470    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn" event={"ID":"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a","Type":"ContainerStarted","Data":"3ca8e9d776e2b14a79b73df1324530045e9cfd59217562775db4c97a67485d07"}
Oct 30 19:57:35.451349 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:35.451320    6777 generic.go:356] "Generic (PLEG): container finished" podID="2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a" containerID="478584481f77451d0debc5fb5ee08a2ef0adba4cbe7db076e8542d0fe8771d4c" exitCode=0
Oct 30 19:57:35.451679 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:35.451402    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn" event={"ID":"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a","Type":"ContainerDied","Data":"478584481f77451d0debc5fb5ee08a2ef0adba4cbe7db076e8542d0fe8771d4c"}
Oct 30 19:57:36.581717 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.581694    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:36.661846 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.661815    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-openshift-ca\") pod \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") "
Oct 30 19:57:36.661995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.661918    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-xks-ca\") pod \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") "
Oct 30 19:57:36.661995 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.661951    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-kube-api-access-t4ltf\" (UniqueName: \"kubernetes.io/projected/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-kube-api-access-t4ltf\") pod \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\" (UID: \"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a\") "
Oct 30 19:57:36.662219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.662196    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-openshift-ca" pod "2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a" (UID: "2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:36.662316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.662300    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:36.663862 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.663836    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-kube-api-access-t4ltf" pod "2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a" (UID: "2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a"). InnerVolumeSpecName "kube-api-access-t4ltf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:36.663965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.663859    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-xks-ca" pod "2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a" (UID: "2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:36.763237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.763183    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4ltf\" (UniqueName: \"kubernetes.io/projected/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-kube-api-access-t4ltf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:36.763237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:36.763206    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:37.462188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:37.462165    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"
Oct 30 19:57:37.462188 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:37.462173    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn" event={"ID":"2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a","Type":"ContainerDied","Data":"3ca8e9d776e2b14a79b73df1324530045e9cfd59217562775db4c97a67485d07"}
Oct 30 19:57:37.462359 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:37.462217    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ca8e9d776e2b14a79b73df1324530045e9cfd59217562775db4c97a67485d07"
Oct 30 19:57:43.636664 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:57:43.636631    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:57:45.753303 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.753275    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"]
Oct 30 19:57:45.753776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.753724    6777 cpu_manager.go:414] "containerMap: removing container" podUID="2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a" containerName="revoke-keys"
Oct 30 19:57:45.753776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.753737    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a" containerName="revoke-keys"
Oct 30 19:57:45.753849 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.753805    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a" containerName="revoke-keys"
Oct 30 19:57:45.757127 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.757108    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.775769 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.775723    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-5310d3-openshift-default-dockercfg-j5mxm\""
Oct 30 19:57:45.779045 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.779018    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"]
Oct 30 19:57:45.836659 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.836632    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-kube-api-access-9j55v\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.836790 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.836682    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-data\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.836920 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.836892    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-certs\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.837069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.836949    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-envoy\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.837069 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.837027    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-istio-token\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.837185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.837093    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-credential-socket\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.837185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.837155    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2778909c-0993-4595-b993-231392e953a4-istiod-ca-cert\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.837290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.837215    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-socket\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.837290 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.837243    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2778909c-0993-4595-b993-231392e953a4-istio-podinfo\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.937961 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.937930    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-certs\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.937983    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-envoy\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938010    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-istio-token\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938037    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-credential-socket\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938115    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2778909c-0993-4595-b993-231392e953a4-istiod-ca-cert\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938160    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-socket\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938200 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938183    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2778909c-0993-4595-b993-231392e953a4-istio-podinfo\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938243    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-kube-api-access-9j55v\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938275    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-data\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938555 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938428    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-certs\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938548    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-socket\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938608    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-credential-socket\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938709 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938638    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-data\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.938854 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.938835    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2778909c-0993-4595-b993-231392e953a4-istiod-ca-cert\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.940483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.940457    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-envoy\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.940685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.940668    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2778909c-0993-4595-b993-231392e953a4-istio-podinfo\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.956827 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.956804    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-kube-api-access-9j55v\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:45.961470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:45.961453    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-istio-token\") pod \"e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") " pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:46.069195 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:46.069172    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:46.352440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:46.352383    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"]
Oct 30 19:57:46.354706 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:57:46.354680    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2778909c_0993_4595_b993_231392e953a4.slice/crio-ebb1b332cb31294ac140aeb5e72063e49123515034e761ba8eb91cdb22769131 WatchSource:0}: Error finding container ebb1b332cb31294ac140aeb5e72063e49123515034e761ba8eb91cdb22769131: Status 404 returned error can't find the container with id ebb1b332cb31294ac140aeb5e72063e49123515034e761ba8eb91cdb22769131
Oct 30 19:57:46.356690 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:46.356664    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:46.356751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:46.356720    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:46.356751 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:46.356747    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:57:46.506732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:46.506699    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" event={"ID":"2778909c-0993-4595-b993-231392e953a4","Type":"ContainerStarted","Data":"4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b"}
Oct 30 19:57:46.506895 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:46.506752    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" event={"ID":"2778909c-0993-4595-b993-231392e953a4","Type":"ContainerStarted","Data":"ebb1b332cb31294ac140aeb5e72063e49123515034e761ba8eb91cdb22769131"}
Oct 30 19:57:46.532346 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:46.532277    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" podStartSLOduration=1.532264929 podStartE2EDuration="1.532264929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:46.53090864 +0000 UTC m=+3048.527824047" watchObservedRunningTime="2026-10-30 19:57:46.532264929 +0000 UTC m=+3048.529180311"
Oct 30 19:57:47.033373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:47.033341    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"]
Oct 30 19:57:47.036339 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:47.036311    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zzjc9"]
Oct 30 19:57:47.069718 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:47.069695    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:47.074189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:47.074168    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:47.512758 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:47.512684    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:47.513716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:47.513689    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:57:48.112159 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:48.112120    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaffb6b1-9d3b-4868-95c6-f7b80674ec77" path="/var/lib/kubelet/pods/eaffb6b1-9d3b-4868-95c6-f7b80674ec77/volumes"
Oct 30 19:57:50.598797 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.598764    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"]
Oct 30 19:57:50.602818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.602798    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:57:50.609592 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.609563    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-137da305-dockercfg-xjrlm\""
Oct 30 19:57:50.612356 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.612333    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5310d3\""
Oct 30 19:57:50.614488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.614465    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"]
Oct 30 19:57:50.679852 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.679830    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aff0978-c8b1-4c36-b7f1-daba4cc28226-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp\" (UID: \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:57:50.679999 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.679979    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmkp4\" (UniqueName: \"kubernetes.io/projected/7aff0978-c8b1-4c36-b7f1-daba4cc28226-kube-api-access-dmkp4\") pod \"payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp\" (UID: \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:57:50.781183 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.781151    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmkp4\" (UniqueName: \"kubernetes.io/projected/7aff0978-c8b1-4c36-b7f1-daba4cc28226-kube-api-access-dmkp4\") pod \"payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp\" (UID: \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:57:50.781366 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.781223    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aff0978-c8b1-4c36-b7f1-daba4cc28226-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp\" (UID: \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:57:50.781912 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.781891    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aff0978-c8b1-4c36-b7f1-daba4cc28226-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp\" (UID: \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:57:50.801066 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.801034    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmkp4\" (UniqueName: \"kubernetes.io/projected/7aff0978-c8b1-4c36-b7f1-daba4cc28226-kube-api-access-dmkp4\") pod \"payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp\" (UID: \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:57:50.917199 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:50.917134    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:57:51.190963 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:51.190938    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"]
Oct 30 19:57:51.192430 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:57:51.192407    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aff0978_c8b1_4c36_b7f1_daba4cc28226.slice/crio-9e390c98e18d7973f689369dd23e71812bb522aeed93ad14e575c9e33bf15187 WatchSource:0}: Error finding container 9e390c98e18d7973f689369dd23e71812bb522aeed93ad14e575c9e33bf15187: Status 404 returned error can't find the container with id 9e390c98e18d7973f689369dd23e71812bb522aeed93ad14e575c9e33bf15187
Oct 30 19:57:51.536036 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:51.536007    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp" event={"ID":"7aff0978-c8b1-4c36-b7f1-daba4cc28226","Type":"ContainerStarted","Data":"7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b"}
Oct 30 19:57:51.536259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:51.536078    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp" event={"ID":"7aff0978-c8b1-4c36-b7f1-daba4cc28226","Type":"ContainerStarted","Data":"9e390c98e18d7973f689369dd23e71812bb522aeed93ad14e575c9e33bf15187"}
Oct 30 19:57:51.536259 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:51.536150    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:57:51.556974 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:51.556916    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp" podStartSLOduration=1.556900755 podStartE2EDuration="1.556900755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:51.556384445 +0000 UTC m=+3053.553299829" watchObservedRunningTime="2026-10-30 19:57:51.556900755 +0000 UTC m=+3053.553816139"
Oct 30 19:57:52.834732 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:57:52.834696    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:58:02.514900 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:02.514868    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Oct 30 19:58:02.515363 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:02.514931    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Oct 30 19:58:02.544775 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:02.544744    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:58:02.619480 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:02.619447    6777 generic.go:356] "Generic (PLEG): container finished" podID="2778909c-0993-4595-b993-231392e953a4" containerID="4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b" exitCode=0
Oct 30 19:58:02.619648 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:02.619533    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" event={"ID":"2778909c-0993-4595-b993-231392e953a4","Type":"ContainerDied","Data":"4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b"}
Oct 30 19:58:02.620184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:02.620161    6777 scope.go:122] "RemoveContainer" containerID="4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b"
Oct 30 19:58:02.621841 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:02.621805    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:02.621938 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:02.621902    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:02.621982 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:02.621955    6777 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:58:03.627177 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:03.627141    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" event={"ID":"2778909c-0993-4595-b993-231392e953a4","Type":"ContainerStarted","Data":"332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe"}
Oct 30 19:58:04.073231 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:04.073191    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:58:04.073604 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:04.073576    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Oct 30 19:58:04.073733 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:04.073636    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Oct 30 19:58:05.072081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:05.071527    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Oct 30 19:58:05.072081 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:05.071593    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Oct 30 19:58:06.046237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:06.046205    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"]
Oct 30 19:58:06.068080 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:06.049483    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2dlq2"]
Oct 30 19:58:06.074482 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:06.074451    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Oct 30 19:58:06.074928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:06.074509    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Oct 30 19:58:06.111146 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:06.111114    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fe8684-0a04-4918-b50c-72cfd54b7db4" path="/var/lib/kubelet/pods/c5fe8684-0a04-4918-b50c-72cfd54b7db4/volumes"
Oct 30 19:58:06.773704 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:06.773673    6777 scope.go:122] "RemoveContainer" containerID="83fbb17dfed1819320b370c4528d9f080a5ff3fdfc8ffc2350e3246264f1fe19"
Oct 30 19:58:06.783106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:06.783082    6777 scope.go:122] "RemoveContainer" containerID="f02aa6443a7d6482ad0f026eb0bed723ab551a3236f804e65bc3fe90736f3228"
Oct 30 19:58:06.791645 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:06.791623    6777 scope.go:122] "RemoveContainer" containerID="6cc4d5cc7e5dad2bcded37acbaa4880ed56bae7197615704cf1457318f89717a"
Oct 30 19:58:06.801929 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:06.801912    6777 scope.go:122] "RemoveContainer" containerID="f5f851bca7360893d053e5bb932bb4c78d8852d161608a288aa32b06d4fe86cd"
Oct 30 19:58:07.073535 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:07.073496    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Oct 30 19:58:07.073730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:07.073559    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Oct 30 19:58:07.985106 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:07.983813    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:58:08.075644 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:08.075614    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Oct 30 19:58:08.075899 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:08.075873    6777 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Oct 30 19:58:09.083651 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:09.083550    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:58:09.084091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:09.083968    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:58:09.084688 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:09.084667    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:58:13.639438 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:58:13.639399    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:58:23.165901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.165864    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:58:23.512607 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.512478    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"]
Oct 30 19:58:23.523905 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.521294    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.532611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.532157    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:58:23.532611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.532298    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:58:23.532611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.532354    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:58:23.532611 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.532440    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:58:23.535171 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.535147    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"]
Oct 30 19:58:23.575360 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.575320    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bf6ba1-plhjg\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.575564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.575372    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bf6ba1-plhjg\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.575564 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.575517    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwwlf\" (UniqueName: \"kubernetes.io/projected/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-kube-api-access-kwwlf\") pod \"maas-api-revoke-sub-98bf6ba1-plhjg\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.676473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.676433    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bf6ba1-plhjg\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.676473 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.676474    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bf6ba1-plhjg\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.676734 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.676515    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwwlf\" (UniqueName: \"kubernetes.io/projected/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-kube-api-access-kwwlf\") pod \"maas-api-revoke-sub-98bf6ba1-plhjg\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.677234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.677213    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bf6ba1-plhjg\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.678880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.678863    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bf6ba1-plhjg\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.689219 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.689193    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwwlf\" (UniqueName: \"kubernetes.io/projected/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-kube-api-access-kwwlf\") pod \"maas-api-revoke-sub-98bf6ba1-plhjg\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:23.842411 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:23.842380    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:24.027675 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:24.027635    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"]
Oct 30 19:58:24.030872 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:24.030832    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-662xx"]
Oct 30 19:58:24.110703 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:24.110619    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3472bece-4e64-4a0d-bab7-b7bb22b6f0ec" path="/var/lib/kubelet/pods/3472bece-4e64-4a0d-bab7-b7bb22b6f0ec/volumes"
Oct 30 19:58:24.117443 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:24.117423    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"]
Oct 30 19:58:24.119367 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:58:24.119344    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a69ba63_c9b6_4b8e_922f_3833098e4dfc.slice/crio-124474ec4ce357efa42f44d5ddc4243cfbecd8a0a9a0a05024834840e8d89df2 WatchSource:0}: Error finding container 124474ec4ce357efa42f44d5ddc4243cfbecd8a0a9a0a05024834840e8d89df2: Status 404 returned error can't find the container with id 124474ec4ce357efa42f44d5ddc4243cfbecd8a0a9a0a05024834840e8d89df2
Oct 30 19:58:24.755310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:24.755221    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg" event={"ID":"0a69ba63-c9b6-4b8e-922f-3833098e4dfc","Type":"ContainerStarted","Data":"8b82ee5a5bad31a0ebce7bf22c50bc762812203575bc587f40754d6d9c1c21df"}
Oct 30 19:58:24.755310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:24.755278    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg" event={"ID":"0a69ba63-c9b6-4b8e-922f-3833098e4dfc","Type":"ContainerStarted","Data":"124474ec4ce357efa42f44d5ddc4243cfbecd8a0a9a0a05024834840e8d89df2"}
Oct 30 19:58:24.779091 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:24.779005    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg" podStartSLOduration=1.778990377 podStartE2EDuration="1.778990377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:24.778719232 +0000 UTC m=+3086.775634617" watchObservedRunningTime="2026-10-30 19:58:24.778990377 +0000 UTC m=+3086.775905756"
Oct 30 19:58:25.762514 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:25.762470    6777 generic.go:356] "Generic (PLEG): container finished" podID="0a69ba63-c9b6-4b8e-922f-3833098e4dfc" containerID="8b82ee5a5bad31a0ebce7bf22c50bc762812203575bc587f40754d6d9c1c21df" exitCode=0
Oct 30 19:58:25.762946 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:25.762565    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg" event={"ID":"0a69ba63-c9b6-4b8e-922f-3833098e4dfc","Type":"ContainerDied","Data":"8b82ee5a5bad31a0ebce7bf22c50bc762812203575bc587f40754d6d9c1c21df"}
Oct 30 19:58:26.900447 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:26.900421    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:27.013479 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.013445    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-xks-ca\") pod \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") "
Oct 30 19:58:27.013699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.013547    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-kube-api-access-kwwlf\" (UniqueName: \"kubernetes.io/projected/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-kube-api-access-kwwlf\") pod \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") "
Oct 30 19:58:27.013699 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.013684    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-openshift-ca\") pod \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\" (UID: \"0a69ba63-c9b6-4b8e-922f-3833098e4dfc\") "
Oct 30 19:58:27.014088 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.014032    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-openshift-ca" pod "0a69ba63-c9b6-4b8e-922f-3833098e4dfc" (UID: "0a69ba63-c9b6-4b8e-922f-3833098e4dfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:27.015517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.015496    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-xks-ca" pod "0a69ba63-c9b6-4b8e-922f-3833098e4dfc" (UID: "0a69ba63-c9b6-4b8e-922f-3833098e4dfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:27.015746 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.015723    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-kube-api-access-kwwlf" pod "0a69ba63-c9b6-4b8e-922f-3833098e4dfc" (UID: "0a69ba63-c9b6-4b8e-922f-3833098e4dfc"). InnerVolumeSpecName "kube-api-access-kwwlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:27.114988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.114951    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwwlf\" (UniqueName: \"kubernetes.io/projected/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-kube-api-access-kwwlf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:27.114988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.114987    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:27.115237 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.114999    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a69ba63-c9b6-4b8e-922f-3833098e4dfc-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:27.782316 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.782279    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"
Oct 30 19:58:27.782530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.782318    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg" event={"ID":"0a69ba63-c9b6-4b8e-922f-3833098e4dfc","Type":"ContainerDied","Data":"124474ec4ce357efa42f44d5ddc4243cfbecd8a0a9a0a05024834840e8d89df2"}
Oct 30 19:58:27.782530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:27.782414    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="124474ec4ce357efa42f44d5ddc4243cfbecd8a0a9a0a05024834840e8d89df2"
Oct 30 19:58:34.401685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.401653    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"]
Oct 30 19:58:34.402162 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.402147    6777 cpu_manager.go:414] "containerMap: removing container" podUID="0a69ba63-c9b6-4b8e-922f-3833098e4dfc" containerName="revoke-keys"
Oct 30 19:58:34.402224 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.402163    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a69ba63-c9b6-4b8e-922f-3833098e4dfc" containerName="revoke-keys"
Oct 30 19:58:34.402274 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.402242    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a69ba63-c9b6-4b8e-922f-3833098e4dfc" containerName="revoke-keys"
Oct 30 19:58:34.405189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.405170    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.412901 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.412808    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:58:34.413147 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.413127    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:58:34.413587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.413525    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:58:34.414185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.414167    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:58:34.419484 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.419457    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"]
Oct 30 19:58:34.547037 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.546984    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:58:34.594530 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.594495    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.594684 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.594554    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwq8t\" (UniqueName: \"kubernetes.io/projected/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-kube-api-access-rwq8t\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.594740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.594707    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.695383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.695289    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.695383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.695362    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.695663 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.695434    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwq8t\" (UniqueName: \"kubernetes.io/projected/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-kube-api-access-rwq8t\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.696168 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.696141    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.697880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.697861    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.708325 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.708302    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwq8t\" (UniqueName: \"kubernetes.io/projected/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-kube-api-access-rwq8t\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:34.724102 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:34.724078    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:35.000787 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:35.000757    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"]
Oct 30 19:58:35.002425 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:58:35.002400    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed42696_e08f_4bc1_b51e_5fb631cdf9ce.slice/crio-71f8263cb6101ba33393060f8fe4570690ccd61dbb03cde91b25cbdbc9385c27 WatchSource:0}: Error finding container 71f8263cb6101ba33393060f8fe4570690ccd61dbb03cde91b25cbdbc9385c27: Status 404 returned error can't find the container with id 71f8263cb6101ba33393060f8fe4570690ccd61dbb03cde91b25cbdbc9385c27
Oct 30 19:58:35.827470 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:35.827425    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6" event={"ID":"bed42696-e08f-4bc1-b51e-5fb631cdf9ce","Type":"ContainerStarted","Data":"780c14f261a38398f6905ad988558aa552ee30838836b883029b5e453609700d"}
Oct 30 19:58:35.827992 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:35.827495    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6" event={"ID":"bed42696-e08f-4bc1-b51e-5fb631cdf9ce","Type":"ContainerStarted","Data":"71f8263cb6101ba33393060f8fe4570690ccd61dbb03cde91b25cbdbc9385c27"}
Oct 30 19:58:35.848096 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:35.848026    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6" podStartSLOduration=1.848009533 podStartE2EDuration="1.848009533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:35.847140368 +0000 UTC m=+3097.844055753" watchObservedRunningTime="2026-10-30 19:58:35.848009533 +0000 UTC m=+3097.844924918"
Oct 30 19:58:36.834777 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:36.834740    6777 generic.go:356] "Generic (PLEG): container finished" podID="bed42696-e08f-4bc1-b51e-5fb631cdf9ce" containerID="780c14f261a38398f6905ad988558aa552ee30838836b883029b5e453609700d" exitCode=0
Oct 30 19:58:36.835207 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:36.834797    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6" event={"ID":"bed42696-e08f-4bc1-b51e-5fb631cdf9ce","Type":"ContainerDied","Data":"780c14f261a38398f6905ad988558aa552ee30838836b883029b5e453609700d"}
Oct 30 19:58:37.974818 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:37.974791    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:38.130771 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.130695    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-xks-ca\") pod \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") "
Oct 30 19:58:38.130771 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.130757    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-kube-api-access-rwq8t\" (UniqueName: \"kubernetes.io/projected/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-kube-api-access-rwq8t\") pod \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") "
Oct 30 19:58:38.131039 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.130855    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-openshift-ca\") pod \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\" (UID: \"bed42696-e08f-4bc1-b51e-5fb631cdf9ce\") "
Oct 30 19:58:38.131289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.131263    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-openshift-ca" pod "bed42696-e08f-4bc1-b51e-5fb631cdf9ce" (UID: "bed42696-e08f-4bc1-b51e-5fb631cdf9ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:38.132740 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.132715    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-xks-ca" pod "bed42696-e08f-4bc1-b51e-5fb631cdf9ce" (UID: "bed42696-e08f-4bc1-b51e-5fb631cdf9ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:38.132848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.132715    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-kube-api-access-rwq8t" pod "bed42696-e08f-4bc1-b51e-5fb631cdf9ce" (UID: "bed42696-e08f-4bc1-b51e-5fb631cdf9ce"). InnerVolumeSpecName "kube-api-access-rwq8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:38.232519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.232491    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:38.232519 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.232518    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:38.232715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.232528    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwq8t\" (UniqueName: \"kubernetes.io/projected/bed42696-e08f-4bc1-b51e-5fb631cdf9ce-kube-api-access-rwq8t\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:38.846736 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.846705    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"
Oct 30 19:58:38.846921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.846715    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6" event={"ID":"bed42696-e08f-4bc1-b51e-5fb631cdf9ce","Type":"ContainerDied","Data":"71f8263cb6101ba33393060f8fe4570690ccd61dbb03cde91b25cbdbc9385c27"}
Oct 30 19:58:38.846921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:38.846826    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f8263cb6101ba33393060f8fe4570690ccd61dbb03cde91b25cbdbc9385c27"
Oct 30 19:58:43.640207 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:58:43.640167    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:58:44.540606 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:44.540559    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"]
Oct 30 19:58:45.232573 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:45.232539    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"]
Oct 30 19:58:45.235483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:45.235457    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5310d3-mkdg6"]
Oct 30 19:58:46.113373 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:46.113338    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed42696-e08f-4bc1-b51e-5fb631cdf9ce" path="/var/lib/kubelet/pods/bed42696-e08f-4bc1-b51e-5fb631cdf9ce/volumes"
Oct 30 19:58:46.384906 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:46.384824    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:58:49.541579 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:49.541529    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp" podUID="7aff0978-c8b1-4c36-b7f1-daba4cc28226" containerName="payload-pre-processing" containerID="cri-o://7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b" gracePeriod=25
Oct 30 19:58:52.543160 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:52.543122    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp" podUID="7aff0978-c8b1-4c36-b7f1-daba4cc28226" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:58:52.918310 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:52.918219    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"]
Oct 30 19:58:52.918517 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:52.918489    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy" containerID="cri-o://332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe" gracePeriod=30
Oct 30 19:58:54.086616 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:54.086572    6777 patch_prober.go:29] interesting pod/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 19:58:54.087046 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:54.086637    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:58:58.174440 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.174416    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:58:58.243279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243200    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2778909c-0993-4595-b993-231392e953a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2778909c-0993-4595-b993-231392e953a4-istio-podinfo\") pod \"2778909c-0993-4595-b993-231392e953a4\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") "
Oct 30 19:58:58.243279 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243237    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2778909c-0993-4595-b993-231392e953a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2778909c-0993-4595-b993-231392e953a4-istiod-ca-cert\") pod \"2778909c-0993-4595-b993-231392e953a4\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") "
Oct 30 19:58:58.243529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243286    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-credential-socket\") pod \"2778909c-0993-4595-b993-231392e953a4\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") "
Oct 30 19:58:58.243529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243346    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-socket\") pod \"2778909c-0993-4595-b993-231392e953a4\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") "
Oct 30 19:58:58.243529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243365    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-data\") pod \"2778909c-0993-4595-b993-231392e953a4\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") "
Oct 30 19:58:58.243529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243401    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-certs\") pod \"2778909c-0993-4595-b993-231392e953a4\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") "
Oct 30 19:58:58.243529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243463    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-istio-token\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-istio-token\") pod \"2778909c-0993-4595-b993-231392e953a4\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") "
Oct 30 19:58:58.243529 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243501    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-kube-api-access-9j55v\") pod \"2778909c-0993-4595-b993-231392e953a4\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") "
Oct 30 19:58:58.243884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243543    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-envoy\") pod \"2778909c-0993-4595-b993-231392e953a4\" (UID: \"2778909c-0993-4595-b993-231392e953a4\") "
Oct 30 19:58:58.243884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243668    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-credential-socket" pod "2778909c-0993-4595-b993-231392e953a4" (UID: "2778909c-0993-4595-b993-231392e953a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:58.243884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243677    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-socket" pod "2778909c-0993-4595-b993-231392e953a4" (UID: "2778909c-0993-4595-b993-231392e953a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:58.243884 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243713    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2778909c-0993-4595-b993-231392e953a4-istiod-ca-cert" pod "2778909c-0993-4595-b993-231392e953a4" (UID: "2778909c-0993-4595-b993-231392e953a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:58.244149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243906    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-certs" pod "2778909c-0993-4595-b993-231392e953a4" (UID: "2778909c-0993-4595-b993-231392e953a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:58.244149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243929    6777 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.244149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243946    6777 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2778909c-0993-4595-b993-231392e953a4-istiod-ca-cert\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.244149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.243955    6777 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-credential-socket\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.244149 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.244014    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-data" pod "2778909c-0993-4595-b993-231392e953a4" (UID: "2778909c-0993-4595-b993-231392e953a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:58.245715 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.245687    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2778909c-0993-4595-b993-231392e953a4-istio-podinfo" pod "2778909c-0993-4595-b993-231392e953a4" (UID: "2778909c-0993-4595-b993-231392e953a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:58.245823 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.245802    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-kube-api-access-9j55v" pod "2778909c-0993-4595-b993-231392e953a4" (UID: "2778909c-0993-4595-b993-231392e953a4"). InnerVolumeSpecName "kube-api-access-9j55v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:58.246163 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.246140    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-envoy" pod "2778909c-0993-4595-b993-231392e953a4" (UID: "2778909c-0993-4595-b993-231392e953a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:58.246460 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.246441    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-istio-token" pod "2778909c-0993-4595-b993-231392e953a4" (UID: "2778909c-0993-4595-b993-231392e953a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:58.344836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.344803    6777 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-data\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.344836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.344832    6777 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-workload-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.344836 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.344841    6777 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-istio-token\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.345194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.344851    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/2778909c-0993-4595-b993-231392e953a4-kube-api-access-9j55v\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.345194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.344862    6777 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2778909c-0993-4595-b993-231392e953a4-istio-envoy\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.345194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.344870    6777 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2778909c-0993-4595-b993-231392e953a4-istio-podinfo\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.370844 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.370814    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:58:58.962634 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.962590    6777 generic.go:356] "Generic (PLEG): container finished" podID="2778909c-0993-4595-b993-231392e953a4" containerID="332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe" exitCode=0
Oct 30 19:58:58.962848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.962730    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" event={"ID":"2778909c-0993-4595-b993-231392e953a4","Type":"ContainerDied","Data":"332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe"}
Oct 30 19:58:58.962848 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.962821    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv" event={"ID":"2778909c-0993-4595-b993-231392e953a4","Type":"ContainerDied","Data":"ebb1b332cb31294ac140aeb5e72063e49123515034e761ba8eb91cdb22769131"}
Oct 30 19:58:58.963020 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.962854    6777 scope.go:122] "RemoveContainer" containerID="332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe"
Oct 30 19:58:58.963128 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.963091    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"
Oct 30 19:58:58.988415 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:58.986835    6777 scope.go:122] "RemoveContainer" containerID="4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b"
Oct 30 19:58:59.001587 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:59.001564    6777 scope.go:122] "RemoveContainer" containerID="332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe"
Oct 30 19:58:59.005386 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:58:59.001955    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe\": container with ID starting with 332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe not found: ID does not exist" containerID="332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe"
Oct 30 19:58:59.005386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:59.001999    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe"} err="failed to get container status \"332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe\": rpc error: code = NotFound desc = could not find container \"332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe\": container with ID starting with 332c77169646a01cd62fa854ea4f3fb4bffdc87f02b458ae35237157a2e7cdfe not found: ID does not exist"
Oct 30 19:58:59.005386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:59.002021    6777 scope.go:122] "RemoveContainer" containerID="4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b"
Oct 30 19:58:59.005386 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:58:59.002389    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b\": container with ID starting with 4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b not found: ID does not exist" containerID="4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b"
Oct 30 19:58:59.005386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:59.002421    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b"} err="failed to get container status \"4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b\": rpc error: code = NotFound desc = could not find container \"4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b\": container with ID starting with 4d6259164b5dded4cf7933b81fe1718c936ff583b0c963d84468d2324d84c16b not found: ID does not exist"
Oct 30 19:58:59.005386 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:59.002719    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"]
Oct 30 19:58:59.008972 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:58:59.008948    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5310d3-openshift-default-5668c5bd4fb4xpv"]
Oct 30 19:59:00.140082 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:00.139663    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2778909c-0993-4595-b993-231392e953a4" path="/var/lib/kubelet/pods/2778909c-0993-4595-b993-231392e953a4/volumes"
Oct 30 19:59:02.543410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:02.543363    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp" podUID="7aff0978-c8b1-4c36-b7f1-daba4cc28226" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:59:06.834410 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:06.834379    6777 scope.go:122] "RemoveContainer" containerID="daa4106a021ddbd89bbf51451bff1cbfca632ee5c7a8e0f874e9e575a82c37ac"
Oct 30 19:59:06.843214 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:06.843195    6777 scope.go:122] "RemoveContainer" containerID="6be4606fbb837fa0989b923d1abc1934a7010a82425620fc62abec558882dca1"
Oct 30 19:59:06.850927 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:06.850910    6777 scope.go:122] "RemoveContainer" containerID="071c276205a49794889050f4a7f4d3d295c2bf9f5cc2d60419f1d1486e7c46ce"
Oct 30 19:59:09.803569 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:09.803492    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:59:09.926289 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:09.926254    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aff0978-c8b1-4c36-b7f1-daba4cc28226-kube-api-access-dmkp4\" (UniqueName: \"kubernetes.io/projected/7aff0978-c8b1-4c36-b7f1-daba4cc28226-kube-api-access-dmkp4\") pod \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\" (UID: \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\") "
Oct 30 19:59:09.926495 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:09.926339    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aff0978-c8b1-4c36-b7f1-daba4cc28226-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aff0978-c8b1-4c36-b7f1-daba4cc28226-plugins-config-volume\") pod \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\" (UID: \"7aff0978-c8b1-4c36-b7f1-daba4cc28226\") "
Oct 30 19:59:09.926730 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:09.926708    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aff0978-c8b1-4c36-b7f1-daba4cc28226-plugins-config-volume" pod "7aff0978-c8b1-4c36-b7f1-daba4cc28226" (UID: "7aff0978-c8b1-4c36-b7f1-daba4cc28226"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:09.928593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:09.928568    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aff0978-c8b1-4c36-b7f1-daba4cc28226-kube-api-access-dmkp4" pod "7aff0978-c8b1-4c36-b7f1-daba4cc28226" (UID: "7aff0978-c8b1-4c36-b7f1-daba4cc28226"). InnerVolumeSpecName "kube-api-access-dmkp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:10.028184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.028140    6777 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aff0978-c8b1-4c36-b7f1-daba4cc28226-plugins-config-volume\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:10.028184 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.028179    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmkp4\" (UniqueName: \"kubernetes.io/projected/7aff0978-c8b1-4c36-b7f1-daba4cc28226-kube-api-access-dmkp4\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:10.084928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.084804    6777 generic.go:356] "Generic (PLEG): container finished" podID="7aff0978-c8b1-4c36-b7f1-daba4cc28226" containerID="7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b" exitCode=0
Oct 30 19:59:10.084928 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.084920    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp" event={"ID":"7aff0978-c8b1-4c36-b7f1-daba4cc28226","Type":"ContainerDied","Data":"7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b"}
Oct 30 19:59:10.085278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.084951    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"
Oct 30 19:59:10.085278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.084974    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp" event={"ID":"7aff0978-c8b1-4c36-b7f1-daba4cc28226","Type":"ContainerDied","Data":"9e390c98e18d7973f689369dd23e71812bb522aeed93ad14e575c9e33bf15187"}
Oct 30 19:59:10.085278 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.085006    6777 scope.go:122] "RemoveContainer" containerID="7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b"
Oct 30 19:59:10.096609 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.096581    6777 scope.go:122] "RemoveContainer" containerID="7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b"
Oct 30 19:59:10.097159 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:59:10.097123    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b\": container with ID starting with 7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b not found: ID does not exist" containerID="7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b"
Oct 30 19:59:10.097280 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.097159    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b"} err="failed to get container status \"7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b\": rpc error: code = NotFound desc = could not find container \"7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b\": container with ID starting with 7fee364abe59a2fd1a1dc449b37151dc8c7856d15a32e2d8de15bdf9e8b5545b not found: ID does not exist"
Oct 30 19:59:10.112172 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.112138    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"]
Oct 30 19:59:10.118380 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:10.118353    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5310d3-6775bzk7zp"]
Oct 30 19:59:12.118716 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:12.118676    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aff0978-c8b1-4c36-b7f1-daba4cc28226" path="/var/lib/kubelet/pods/7aff0978-c8b1-4c36-b7f1-daba4cc28226/volumes"
Oct 30 19:59:13.638763 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:59:13.638725    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:59:14.437015 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.436981    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"]
Oct 30 19:59:14.437509 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437494    6777 cpu_manager.go:414] "containerMap: removing container" podUID="7aff0978-c8b1-4c36-b7f1-daba4cc28226" containerName="payload-pre-processing"
Oct 30 19:59:14.437559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437511    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aff0978-c8b1-4c36-b7f1-daba4cc28226" containerName="payload-pre-processing"
Oct 30 19:59:14.437559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437529    6777 cpu_manager.go:414] "containerMap: removing container" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy"
Oct 30 19:59:14.437559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437534    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy"
Oct 30 19:59:14.437559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437545    6777 cpu_manager.go:414] "containerMap: removing container" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy"
Oct 30 19:59:14.437559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437550    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy"
Oct 30 19:59:14.437559 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437557    6777 cpu_manager.go:414] "containerMap: removing container" podUID="bed42696-e08f-4bc1-b51e-5fb631cdf9ce" containerName="revoke-keys"
Oct 30 19:59:14.437767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437562    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed42696-e08f-4bc1-b51e-5fb631cdf9ce" containerName="revoke-keys"
Oct 30 19:59:14.437767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437620    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aff0978-c8b1-4c36-b7f1-daba4cc28226" containerName="payload-pre-processing"
Oct 30 19:59:14.437767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437629    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy"
Oct 30 19:59:14.437767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437636    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="2778909c-0993-4595-b993-231392e953a4" containerName="istio-proxy"
Oct 30 19:59:14.437767 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.437642    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed42696-e08f-4bc1-b51e-5fb631cdf9ce" containerName="revoke-keys"
Oct 30 19:59:14.441893 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.441875    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.450487 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.450454    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0ba273d9-kserve-self-signed-certs\""
Oct 30 19:59:14.451642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.451608    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"]
Oct 30 19:59:14.569329 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.569295    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8xm\" (UniqueName: \"kubernetes.io/projected/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-kube-api-access-bx8xm\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.569513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.569350    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-home\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.569513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.569457    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tmp-dir\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.569513 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.569485    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-dshm\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.569685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.569658    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tls-certs\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.569745 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.569701    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-model-cache\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.671122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.671087    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tls-certs\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.671122 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.671129    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-model-cache\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.671650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.671180    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8xm\" (UniqueName: \"kubernetes.io/projected/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-kube-api-access-bx8xm\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.671650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.671216    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-home\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.671650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.671255    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tmp-dir\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.671650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.671373    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-dshm\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.671650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.671630    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-model-cache\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.671650 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.671648    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tmp-dir\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.671943 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.671728    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-home\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.673474 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.673453    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-dshm\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.673593 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.673558    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tls-certs\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.691546 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.691482    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8xm\" (UniqueName: \"kubernetes.io/projected/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-kube-api-access-bx8xm\") pod \"e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") " pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:14.763194 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:14.763169    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:15.035297 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:15.035276    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"]
Oct 30 19:59:15.037883 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:59:15.037857    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcc6d9d4_3d8a_4461_9505_fb2c17fa1faa.slice/crio-d8c5c22ee77625bf041e16072bed4324f6f228e6135251ca517e8dd28ee7b383 WatchSource:0}: Error finding container d8c5c22ee77625bf041e16072bed4324f6f228e6135251ca517e8dd28ee7b383: Status 404 returned error can't find the container with id d8c5c22ee77625bf041e16072bed4324f6f228e6135251ca517e8dd28ee7b383
Oct 30 19:59:15.113860 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:15.113832    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg" event={"ID":"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa","Type":"ContainerStarted","Data":"93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d"}
Oct 30 19:59:15.113960 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:15.113871    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg" event={"ID":"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa","Type":"ContainerStarted","Data":"d8c5c22ee77625bf041e16072bed4324f6f228e6135251ca517e8dd28ee7b383"}
Oct 30 19:59:15.136196 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:15.136154    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg" podStartSLOduration=1.136140018 podStartE2EDuration="1.136140018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:15.134427433 +0000 UTC m=+3137.131342821" watchObservedRunningTime="2026-10-30 19:59:15.136140018 +0000 UTC m=+3137.133055402"
Oct 30 19:59:15.764123 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:15.764093    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:15.877642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:15.877612    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:59:24.763613 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:24.763577    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:24.775676 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:24.775645    6777 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:25.216731 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:25.216707    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:31.088336 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.088302    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"]
Oct 30 19:59:31.091880 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.091862    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.100891 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.100867    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"]
Oct 30 19:59:31.101270 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.101249    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:59:31.101374 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.101326    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:59:31.101444 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.101432    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:59:31.101504 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.101481    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:59:31.219933 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.219911    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nb5qd\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.220077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.219974    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nb5qd\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.220077 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.220033    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6pfd\" (UniqueName: \"kubernetes.io/projected/92654488-bdba-40b5-b9a7-4dd95658f441-kube-api-access-l6pfd\") pod \"maas-api-revoke-sub-452b6d7b-nb5qd\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.321140 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.321114    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nb5qd\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.321246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.321143    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pfd\" (UniqueName: \"kubernetes.io/projected/92654488-bdba-40b5-b9a7-4dd95658f441-kube-api-access-l6pfd\") pod \"maas-api-revoke-sub-452b6d7b-nb5qd\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.321246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.321199    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nb5qd\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.321776 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.321753    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nb5qd\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.323441 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.323418    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nb5qd\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.335488 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.335462    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pfd\" (UniqueName: \"kubernetes.io/projected/92654488-bdba-40b5-b9a7-4dd95658f441-kube-api-access-l6pfd\") pod \"maas-api-revoke-sub-452b6d7b-nb5qd\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.354685 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.354625    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:59:31.406406 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.406386    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:31.682737 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:31.682688    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"]
Oct 30 19:59:31.685175 ip-10-0-132-19 kubenswrapper[6777]: W1030 19:59:31.685070    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92654488_bdba_40b5_b9a7_4dd95658f441.slice/crio-9c9cf6d3ca8e7ff152bca4538f397fecde6803fcc34bd47652fe4073c81c51f4 WatchSource:0}: Error finding container 9c9cf6d3ca8e7ff152bca4538f397fecde6803fcc34bd47652fe4073c81c51f4: Status 404 returned error can't find the container with id 9c9cf6d3ca8e7ff152bca4538f397fecde6803fcc34bd47652fe4073c81c51f4
Oct 30 19:59:32.247368 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:32.247277    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd" event={"ID":"92654488-bdba-40b5-b9a7-4dd95658f441","Type":"ContainerStarted","Data":"efbe59a48537520ef5c2e1aeea2637e0a3b6b0c89ea8b58bf2fa1dab54926682"}
Oct 30 19:59:32.247967 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:32.247934    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd" event={"ID":"92654488-bdba-40b5-b9a7-4dd95658f441","Type":"ContainerStarted","Data":"9c9cf6d3ca8e7ff152bca4538f397fecde6803fcc34bd47652fe4073c81c51f4"}
Oct 30 19:59:32.268490 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:32.268436    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd" podStartSLOduration=1.268419087 podStartE2EDuration="1.268419087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:32.266450085 +0000 UTC m=+3154.263365473" watchObservedRunningTime="2026-10-30 19:59:32.268419087 +0000 UTC m=+3154.265334472"
Oct 30 19:59:33.267052 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:33.267017    6777 generic.go:356] "Generic (PLEG): container finished" podID="92654488-bdba-40b5-b9a7-4dd95658f441" containerID="efbe59a48537520ef5c2e1aeea2637e0a3b6b0c89ea8b58bf2fa1dab54926682" exitCode=0
Oct 30 19:59:33.267725 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:33.267699    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd" event={"ID":"92654488-bdba-40b5-b9a7-4dd95658f441","Type":"ContainerDied","Data":"efbe59a48537520ef5c2e1aeea2637e0a3b6b0c89ea8b58bf2fa1dab54926682"}
Oct 30 19:59:35.063085 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.053633    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:35.232921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.232819    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-xks-ca\") pod \"92654488-bdba-40b5-b9a7-4dd95658f441\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") "
Oct 30 19:59:35.232921 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.232889    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-openshift-ca\") pod \"92654488-bdba-40b5-b9a7-4dd95658f441\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") "
Oct 30 19:59:35.233212 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.232990    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92654488-bdba-40b5-b9a7-4dd95658f441-kube-api-access-l6pfd\" (UniqueName: \"kubernetes.io/projected/92654488-bdba-40b5-b9a7-4dd95658f441-kube-api-access-l6pfd\") pod \"92654488-bdba-40b5-b9a7-4dd95658f441\" (UID: \"92654488-bdba-40b5-b9a7-4dd95658f441\") "
Oct 30 19:59:35.233353 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.233322    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-openshift-ca" pod "92654488-bdba-40b5-b9a7-4dd95658f441" (UID: "92654488-bdba-40b5-b9a7-4dd95658f441"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:35.233925 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.233899    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:35.235700 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.235667    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-xks-ca" pod "92654488-bdba-40b5-b9a7-4dd95658f441" (UID: "92654488-bdba-40b5-b9a7-4dd95658f441"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:35.235894 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.235866    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92654488-bdba-40b5-b9a7-4dd95658f441-kube-api-access-l6pfd" pod "92654488-bdba-40b5-b9a7-4dd95658f441" (UID: "92654488-bdba-40b5-b9a7-4dd95658f441"). InnerVolumeSpecName "kube-api-access-l6pfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:35.295520 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.295493    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd"
Oct 30 19:59:35.295649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.295541    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nb5qd" event={"ID":"92654488-bdba-40b5-b9a7-4dd95658f441","Type":"ContainerDied","Data":"9c9cf6d3ca8e7ff152bca4538f397fecde6803fcc34bd47652fe4073c81c51f4"}
Oct 30 19:59:35.295649 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.295612    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c9cf6d3ca8e7ff152bca4538f397fecde6803fcc34bd47652fe4073c81c51f4"
Oct 30 19:59:35.335251 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.335225    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92654488-bdba-40b5-b9a7-4dd95658f441-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:35.335384 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:35.335255    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6pfd\" (UniqueName: \"kubernetes.io/projected/92654488-bdba-40b5-b9a7-4dd95658f441-kube-api-access-l6pfd\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:42.952922 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:42.952894    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"]
Oct 30 19:59:43.333682 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:43.333650    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:59:43.638534 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:59:43.638458    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 19:59:44.663866 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.663833    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"]
Oct 30 19:59:44.665189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.665161    6777 cpu_manager.go:414] "containerMap: removing container" podUID="92654488-bdba-40b5-b9a7-4dd95658f441" containerName="revoke-keys"
Oct 30 19:59:44.665189 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.665188    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92654488-bdba-40b5-b9a7-4dd95658f441" containerName="revoke-keys"
Oct 30 19:59:44.665419 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.665320    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="92654488-bdba-40b5-b9a7-4dd95658f441" containerName="revoke-keys"
Oct 30 19:59:44.669308 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.669285    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.675803 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.675778    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"]
Oct 30 19:59:44.678215 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.676981    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:59:44.680614 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.680593    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:59:44.680831 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.680807    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:59:44.681185 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.681167    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:59:44.738098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.735937    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-288vz\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.738098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.735990    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxcf\" (UniqueName: \"kubernetes.io/projected/896971d6-9846-492a-b434-08ee4cf45b32-kube-api-access-fxxcf\") pod \"maas-api-revoke-sub-3e31a506-288vz\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.738098 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.736151    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-288vz\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.837458 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.837418    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-288vz\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.837646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.837476    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxcf\" (UniqueName: \"kubernetes.io/projected/896971d6-9846-492a-b434-08ee4cf45b32-kube-api-access-fxxcf\") pod \"maas-api-revoke-sub-3e31a506-288vz\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.837646 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.837528    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-288vz\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.838369 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.838340    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-288vz\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.840642 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.840614    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-288vz\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.865299 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.865117    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxcf\" (UniqueName: \"kubernetes.io/projected/896971d6-9846-492a-b434-08ee4cf45b32-kube-api-access-fxxcf\") pod \"maas-api-revoke-sub-3e31a506-288vz\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:44.995130 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:44.994286    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:45.835083 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:45.814238    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"]
Oct 30 19:59:46.382843 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:46.382805    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz" event={"ID":"896971d6-9846-492a-b434-08ee4cf45b32","Type":"ContainerStarted","Data":"6d8eb367cda510bd3486fe8d9e8b73f4eb2e1e1000ab3348eb8d73c507a9c911"}
Oct 30 19:59:47.198239 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:47.198200    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:59:47.407770 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:47.407682    6777 generic.go:356] "Generic (PLEG): container finished" podID="896971d6-9846-492a-b434-08ee4cf45b32" containerID="5ac0f842235e576417ded11b08a2e018c9978b2edf5568b3eef8766e5d269391" exitCode=0
Oct 30 19:59:47.408234 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:47.408191    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz" event={"ID":"896971d6-9846-492a-b434-08ee4cf45b32","Type":"ContainerDied","Data":"5ac0f842235e576417ded11b08a2e018c9978b2edf5568b3eef8766e5d269391"}
Oct 30 19:59:48.819515 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:48.819221    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:48.860092 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:48.856371    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-openshift-ca\") pod \"896971d6-9846-492a-b434-08ee4cf45b32\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") "
Oct 30 19:59:48.871191 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:48.871158    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896971d6-9846-492a-b434-08ee4cf45b32-kube-api-access-fxxcf\" (UniqueName: \"kubernetes.io/projected/896971d6-9846-492a-b434-08ee4cf45b32-kube-api-access-fxxcf\") pod \"896971d6-9846-492a-b434-08ee4cf45b32\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") "
Oct 30 19:59:48.881293 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:48.881266    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-xks-ca\") pod \"896971d6-9846-492a-b434-08ee4cf45b32\" (UID: \"896971d6-9846-492a-b434-08ee4cf45b32\") "
Oct 30 19:59:48.895383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:48.862411    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-openshift-ca" pod "896971d6-9846-492a-b434-08ee4cf45b32" (UID: "896971d6-9846-492a-b434-08ee4cf45b32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:48.916089 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:48.900496    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:48.925113 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:48.906563    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-xks-ca" pod "896971d6-9846-492a-b434-08ee4cf45b32" (UID: "896971d6-9846-492a-b434-08ee4cf45b32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:48.943584 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:48.943543    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896971d6-9846-492a-b434-08ee4cf45b32-kube-api-access-fxxcf" pod "896971d6-9846-492a-b434-08ee4cf45b32" (UID: "896971d6-9846-492a-b434-08ee4cf45b32"). InnerVolumeSpecName "kube-api-access-fxxcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:49.035221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:49.034050    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896971d6-9846-492a-b434-08ee4cf45b32-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:49.035221 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:49.035146    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxcf\" (UniqueName: \"kubernetes.io/projected/896971d6-9846-492a-b434-08ee4cf45b32-kube-api-access-fxxcf\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:49.447798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:49.447689    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz" event={"ID":"896971d6-9846-492a-b434-08ee4cf45b32","Type":"ContainerDied","Data":"6d8eb367cda510bd3486fe8d9e8b73f4eb2e1e1000ab3348eb8d73c507a9c911"}
Oct 30 19:59:49.447798 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:49.447754    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d8eb367cda510bd3486fe8d9e8b73f4eb2e1e1000ab3348eb8d73c507a9c911"
Oct 30 19:59:49.448053 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:49.447821    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-288vz"
Oct 30 19:59:55.460757 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:55.460722    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 19:59:57.973965 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:57.973926    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg" podUID="fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" containerName="main" containerID="cri-o://93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d" gracePeriod=45
Oct 30 19:59:58.280324 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.280299    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:58.396572 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.396537    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-kube-api-access-bx8xm\" (UniqueName: \"kubernetes.io/projected/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-kube-api-access-bx8xm\") pod \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") "
Oct 30 19:59:58.396774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.396624    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-model-cache\") pod \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") "
Oct 30 19:59:58.396774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.396692    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-home\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-home\") pod \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") "
Oct 30 19:59:58.396774 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.396721    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tmp-dir\") pod \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") "
Oct 30 19:59:58.396988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.396777    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-dshm\") pod \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") "
Oct 30 19:59:58.396988 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.396845    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tls-certs\") pod \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\" (UID: \"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa\") "
Oct 30 19:59:58.397154 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.397125    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-home" pod "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" (UID: "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:58.397371 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.397330    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-model-cache" pod "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" (UID: "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:58.397539 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.397409    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tmp-dir" pod "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" (UID: "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:58.400383 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.400346    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-dshm" pod "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" (UID: "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:58.400483 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.400410    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-kube-api-access-bx8xm" pod "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" (UID: "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa"). InnerVolumeSpecName "kube-api-access-bx8xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:58.400701 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.400674    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tls-certs" pod "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" (UID: "fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:58.498228 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.498197    6777 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tls-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:58.498397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.498231    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8xm\" (UniqueName: \"kubernetes.io/projected/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-kube-api-access-bx8xm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:58.498397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.498248    6777 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-model-cache\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:58.498397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.498260    6777 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-home\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:58.498397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.498297    6777 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-tmp-dir\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:58.498397 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.498310    6777 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa-dshm\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:58.528481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.528399    6777 generic.go:356] "Generic (PLEG): container finished" podID="fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" containerID="93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d" exitCode=0
Oct 30 19:59:58.528481 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.528459    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"
Oct 30 19:59:58.528687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.528492    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg" event={"ID":"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa","Type":"ContainerDied","Data":"93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d"}
Oct 30 19:59:58.528687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.528554    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg" event={"ID":"fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa","Type":"ContainerDied","Data":"d8c5c22ee77625bf041e16072bed4324f6f228e6135251ca517e8dd28ee7b383"}
Oct 30 19:59:58.528687 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.528580    6777 scope.go:122] "RemoveContainer" containerID="93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d"
Oct 30 19:59:58.544726 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.544700    6777 scope.go:122] "RemoveContainer" containerID="93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d"
Oct 30 19:59:58.545135 ip-10-0-132-19 kubenswrapper[6777]: E1030 19:59:58.545105    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d\": container with ID starting with 93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d not found: ID does not exist" containerID="93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d"
Oct 30 19:59:58.545246 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.545146    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d"} err="failed to get container status \"93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d\": rpc error: code = NotFound desc = could not find container \"93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d\": container with ID starting with 93c9a6cc80679664a130c566b39f437828944a6c1b77655a19a1f766fa8d9b2d not found: ID does not exist"
Oct 30 19:59:58.560621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.557714    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"]
Oct 30 19:59:58.560621 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:58.560577    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg"]
Oct 30 19:59:59.214276 ip-10-0-132-19 kubenswrapper[6777]: I1030 19:59:59.214237    6777 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-0ba273d9-kserve-5c8584bb45-cqpdg" podUID="fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" containerName="main" probeResult="failure" output="Get \"https://10.134.1.61:8000/ready\": context deadline exceeded"
Oct 30 20:00:00.154775 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.154733    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" path="/var/lib/kubelet/pods/fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa/volumes"
Oct 30 20:00:00.155560 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.155526    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"]
Oct 30 20:00:00.156509 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.156490    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" containerName="main"
Oct 30 20:00:00.156662 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.156647    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" containerName="main"
Oct 30 20:00:00.156817 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.156804    6777 cpu_manager.go:414] "containerMap: removing container" podUID="896971d6-9846-492a-b434-08ee4cf45b32" containerName="revoke-keys"
Oct 30 20:00:00.156942 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.156928    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896971d6-9846-492a-b434-08ee4cf45b32" containerName="revoke-keys"
Oct 30 20:00:00.157371 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.157354    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc6d9d4-3d8a-4461-9505-fb2c17fa1faa" containerName="main"
Oct 30 20:00:00.169083 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.157510    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="896971d6-9846-492a-b434-08ee4cf45b32" containerName="revoke-keys"
Oct 30 20:00:00.179283 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.179262    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"]
Oct 30 20:00:00.179451 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.179393    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:00.194149 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.194119    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:00.200845 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.200826    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:00.201161 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.201137    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 20:00:00.355685 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.355644    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f45g\" (UniqueName: \"kubernetes.io/projected/6201bdc0-f971-426d-bb81-d3bde264db96-kube-api-access-2f45g\") pod \"maas-api-key-cleanup-29889840-87n4s\" (UID: \"6201bdc0-f971-426d-bb81-d3bde264db96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:00.363538 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.363438    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6201bdc0-f971-426d-bb81-d3bde264db96-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-87n4s\" (UID: \"6201bdc0-f971-426d-bb81-d3bde264db96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:00.511128 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.503639    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f45g\" (UniqueName: \"kubernetes.io/projected/6201bdc0-f971-426d-bb81-d3bde264db96-kube-api-access-2f45g\") pod \"maas-api-key-cleanup-29889840-87n4s\" (UID: \"6201bdc0-f971-426d-bb81-d3bde264db96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:00.511128 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.503709    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6201bdc0-f971-426d-bb81-d3bde264db96-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-87n4s\" (UID: \"6201bdc0-f971-426d-bb81-d3bde264db96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:00.534081 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.515150    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6201bdc0-f971-426d-bb81-d3bde264db96-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-87n4s\" (UID: \"6201bdc0-f971-426d-bb81-d3bde264db96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:00.585084 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.559783    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 20:00:00.585286 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.568361    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f45g\" (UniqueName: \"kubernetes.io/projected/6201bdc0-f971-426d-bb81-d3bde264db96-kube-api-access-2f45g\") pod \"maas-api-key-cleanup-29889840-87n4s\" (UID: \"6201bdc0-f971-426d-bb81-d3bde264db96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:00.767133 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:00.766928    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:01.166243 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:01.166217    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"]
Oct 30 20:00:01.168843 ip-10-0-132-19 kubenswrapper[6777]: W1030 20:00:01.168812    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6201bdc0_f971_426d_bb81_d3bde264db96.slice/crio-b2d4686732a07e280353fe357fc897795d0027afcf9dca880a0c0f71f7ec9206 WatchSource:0}: Error finding container b2d4686732a07e280353fe357fc897795d0027afcf9dca880a0c0f71f7ec9206: Status 404 returned error can't find the container with id b2d4686732a07e280353fe357fc897795d0027afcf9dca880a0c0f71f7ec9206
Oct 30 20:00:01.625869 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:01.625838    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" event={"ID":"6201bdc0-f971-426d-bb81-d3bde264db96","Type":"ContainerStarted","Data":"b166d0c7d438210e5a390805b48ef520442e6a818e06353c367e3917e721c279"}
Oct 30 20:00:01.626297 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:01.625887    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" event={"ID":"6201bdc0-f971-426d-bb81-d3bde264db96","Type":"ContainerStarted","Data":"b2d4686732a07e280353fe357fc897795d0027afcf9dca880a0c0f71f7ec9206"}
Oct 30 20:00:01.648792 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:01.648738    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" podStartSLOduration=1.6487244840000002 podStartE2EDuration="1.648724484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.647118043 +0000 UTC m=+3183.644033430" watchObservedRunningTime="2026-10-30 20:00:01.648724484 +0000 UTC m=+3183.645639868"
Oct 30 20:00:02.632558 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:02.632519    6777 generic.go:356] "Generic (PLEG): container finished" podID="6201bdc0-f971-426d-bb81-d3bde264db96" containerID="b166d0c7d438210e5a390805b48ef520442e6a818e06353c367e3917e721c279" exitCode=60
Oct 30 20:00:02.632558 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:02.632557    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" event={"ID":"6201bdc0-f971-426d-bb81-d3bde264db96","Type":"ContainerDied","Data":"b166d0c7d438210e5a390805b48ef520442e6a818e06353c367e3917e721c279"}
Oct 30 20:00:02.633020 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:02.632831    6777 scope.go:122] "RemoveContainer" containerID="b166d0c7d438210e5a390805b48ef520442e6a818e06353c367e3917e721c279"
Oct 30 20:00:03.639626 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:03.639587    6777 generic.go:356] "Generic (PLEG): container finished" podID="6201bdc0-f971-426d-bb81-d3bde264db96" containerID="7d277dd4b8ac1b4f8ed9b34bcda2c04f198f3849cf1b51911ffffe7d24c05489" exitCode=60
Oct 30 20:00:03.640118 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:03.639653    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" event={"ID":"6201bdc0-f971-426d-bb81-d3bde264db96","Type":"ContainerDied","Data":"7d277dd4b8ac1b4f8ed9b34bcda2c04f198f3849cf1b51911ffffe7d24c05489"}
Oct 30 20:00:03.640118 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:03.639737    6777 scope.go:122] "RemoveContainer" containerID="b166d0c7d438210e5a390805b48ef520442e6a818e06353c367e3917e721c279"
Oct 30 20:00:03.640118 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:03.639989    6777 scope.go:122] "RemoveContainer" containerID="7d277dd4b8ac1b4f8ed9b34bcda2c04f198f3849cf1b51911ffffe7d24c05489"
Oct 30 20:00:03.640359 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:00:03.640336    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-87n4s_odh-ai-gateway-infra(6201bdc0-f971-426d-bb81-d3bde264db96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" podUID="6201bdc0-f971-426d-bb81-d3bde264db96"
Oct 30 20:00:04.646247 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:04.646218    6777 scope.go:122] "RemoveContainer" containerID="7d277dd4b8ac1b4f8ed9b34bcda2c04f198f3849cf1b51911ffffe7d24c05489"
Oct 30 20:00:04.646634 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:00:04.646431    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-87n4s_odh-ai-gateway-infra(6201bdc0-f971-426d-bb81-d3bde264db96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" podUID="6201bdc0-f971-426d-bb81-d3bde264db96"
Oct 30 20:00:06.882754 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:06.882718    6777 scope.go:122] "RemoveContainer" containerID="7739e3baf7828c48346fa43512457183932575369d0d5c0c84ed9ffc36b4c466"
Oct 30 20:00:06.891684 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:06.891665    6777 scope.go:122] "RemoveContainer" containerID="2e557200d7e5768a730c2dc82f2530828283b21f15aa5f1a8071cac473b15427"
Oct 30 20:00:06.899327 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:06.899307    6777 scope.go:122] "RemoveContainer" containerID="dce7fc87b4d8637dff44f36b4471805420e02812302b4a8be6b1019851715400"
Oct 30 20:00:07.025719 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:07.025689    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"]
Oct 30 20:00:07.029349 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:07.029319    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71f2280-wmwxd"]
Oct 30 20:00:08.110877 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:08.110843    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e36c66-49a4-4ce0-85f0-12ae0872c327" path="/var/lib/kubelet/pods/24e36c66-49a4-4ce0-85f0-12ae0872c327/volumes"
Oct 30 20:00:13.639158 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:00:13.639120    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 20:00:16.868642 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:16.868607    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 20:00:18.108178 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.108149    6777 scope.go:122] "RemoveContainer" containerID="7d277dd4b8ac1b4f8ed9b34bcda2c04f198f3849cf1b51911ffffe7d24c05489"
Oct 30 20:00:18.534494 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.533980    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"]
Oct 30 20:00:18.539185 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.539159    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.547719 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.547694    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"]
Oct 30 20:00:18.552187 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.552152    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f8msk\""
Oct 30 20:00:18.552187 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.552175    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:00:18.552368 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.552246    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 20:00:18.600255 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.600231    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-maas-api-tls\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.600387 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.600276    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-openshift-service-ca\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.600469 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.600441    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcfjv\" (UniqueName: \"kubernetes.io/projected/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-kube-api-access-jcfjv\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.600535 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.600482    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-metrics-certs\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.701611 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.701555    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcfjv\" (UniqueName: \"kubernetes.io/projected/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-kube-api-access-jcfjv\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.701611 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.701587    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-metrics-certs\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.701746 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.701627    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-maas-api-tls\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.701746 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.701655    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-openshift-service-ca\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.702326 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.702302    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-openshift-service-ca\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.704016 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.703992    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-maas-api-tls\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.704142 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.704114    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-metrics-certs\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.713814 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.713793    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcfjv\" (UniqueName: \"kubernetes.io/projected/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-kube-api-access-jcfjv\") pod \"maas-api-5c48877c75-l7zwb\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:18.723900 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.723875    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" event={"ID":"6201bdc0-f971-426d-bb81-d3bde264db96","Type":"ContainerStarted","Data":"e1e01a47116ff3a306947819a9e6f5c3e901fc111b3d0976d2e63988977d3a5c"}
Oct 30 20:00:18.855534 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:18.855509    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:19.130206 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.130182    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"]
Oct 30 20:00:19.130870 ip-10-0-132-19 kubenswrapper[6777]: W1030 20:00:19.130844    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2ffdcc_022d_42b7_9a3c_baeefd2bdb2b.slice/crio-16d6d2b36c6215f0ad3c74c06dbea91ce1038868528d187fe625f98e0aa59cf9 WatchSource:0}: Error finding container 16d6d2b36c6215f0ad3c74c06dbea91ce1038868528d187fe625f98e0aa59cf9: Status 404 returned error can't find the container with id 16d6d2b36c6215f0ad3c74c06dbea91ce1038868528d187fe625f98e0aa59cf9
Oct 30 20:00:19.132354 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.132335    6777 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:00:19.730430 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.730394    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb" event={"ID":"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b","Type":"ContainerStarted","Data":"e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac"}
Oct 30 20:00:19.730617 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.730446    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb" event={"ID":"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b","Type":"ContainerStarted","Data":"16d6d2b36c6215f0ad3c74c06dbea91ce1038868528d187fe625f98e0aa59cf9"}
Oct 30 20:00:19.730617 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.730571    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:19.732252 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.732227    6777 generic.go:356] "Generic (PLEG): container finished" podID="6201bdc0-f971-426d-bb81-d3bde264db96" containerID="e1e01a47116ff3a306947819a9e6f5c3e901fc111b3d0976d2e63988977d3a5c" exitCode=60
Oct 30 20:00:19.732375 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.732273    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" event={"ID":"6201bdc0-f971-426d-bb81-d3bde264db96","Type":"ContainerDied","Data":"e1e01a47116ff3a306947819a9e6f5c3e901fc111b3d0976d2e63988977d3a5c"}
Oct 30 20:00:19.732375 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.732311    6777 scope.go:122] "RemoveContainer" containerID="7d277dd4b8ac1b4f8ed9b34bcda2c04f198f3849cf1b51911ffffe7d24c05489"
Oct 30 20:00:19.732605 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.732588    6777 scope.go:122] "RemoveContainer" containerID="e1e01a47116ff3a306947819a9e6f5c3e901fc111b3d0976d2e63988977d3a5c"
Oct 30 20:00:19.732852 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:00:19.732823    6777 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-87n4s_odh-ai-gateway-infra(6201bdc0-f971-426d-bb81-d3bde264db96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" podUID="6201bdc0-f971-426d-bb81-d3bde264db96"
Oct 30 20:00:19.752980 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.752921    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb" podStartSLOduration=1.345957765 podStartE2EDuration="1.752904159s" totalImagesPullingTime="406.946394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:19.751127176 +0000 UTC m=+3201.748042559" watchObservedRunningTime="2026-10-30 20:00:19.752904159 +0000 UTC m=+3201.749819545"
Oct 30 20:00:19.754929 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:19.754905    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"]
Oct 30 20:00:20.872316 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:20.872286    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:20.922138 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:20.922112    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6201bdc0-f971-426d-bb81-d3bde264db96-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6201bdc0-f971-426d-bb81-d3bde264db96-maas-api-service-ca\") pod \"6201bdc0-f971-426d-bb81-d3bde264db96\" (UID: \"6201bdc0-f971-426d-bb81-d3bde264db96\") "
Oct 30 20:00:20.922301 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:20.922215    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6201bdc0-f971-426d-bb81-d3bde264db96-kube-api-access-2f45g\" (UniqueName: \"kubernetes.io/projected/6201bdc0-f971-426d-bb81-d3bde264db96-kube-api-access-2f45g\") pod \"6201bdc0-f971-426d-bb81-d3bde264db96\" (UID: \"6201bdc0-f971-426d-bb81-d3bde264db96\") "
Oct 30 20:00:20.922582 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:20.922556    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6201bdc0-f971-426d-bb81-d3bde264db96-maas-api-service-ca" pod "6201bdc0-f971-426d-bb81-d3bde264db96" (UID: "6201bdc0-f971-426d-bb81-d3bde264db96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:20.924347 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:20.924324    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6201bdc0-f971-426d-bb81-d3bde264db96-kube-api-access-2f45g" pod "6201bdc0-f971-426d-bb81-d3bde264db96" (UID: "6201bdc0-f971-426d-bb81-d3bde264db96"). InnerVolumeSpecName "kube-api-access-2f45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.024091 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:21.024002    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f45g\" (UniqueName: \"kubernetes.io/projected/6201bdc0-f971-426d-bb81-d3bde264db96-kube-api-access-2f45g\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.024091 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:21.024029    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6201bdc0-f971-426d-bb81-d3bde264db96-maas-api-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.744817 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:21.744791    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"
Oct 30 20:00:21.744980 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:21.744791    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s" event={"ID":"6201bdc0-f971-426d-bb81-d3bde264db96","Type":"ContainerDied","Data":"b2d4686732a07e280353fe357fc897795d0027afcf9dca880a0c0f71f7ec9206"}
Oct 30 20:00:21.744980 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:21.744931    6777 scope.go:122] "RemoveContainer" containerID="e1e01a47116ff3a306947819a9e6f5c3e901fc111b3d0976d2e63988977d3a5c"
Oct 30 20:00:21.767916 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:21.767889    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"]
Oct 30 20:00:21.775832 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:21.775805    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-87n4s"]
Oct 30 20:00:22.111782 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:22.111748    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" path="/var/lib/kubelet/pods/6201bdc0-f971-426d-bb81-d3bde264db96/volumes"
Oct 30 20:00:25.744828 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:25.744802    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:26.752318 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:26.752282    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 20:00:31.875625 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:31.875588    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 20:00:33.727082 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:33.705589    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"]
Oct 30 20:00:33.740513 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:33.740460    6777 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb" podUID="fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" containerName="maas-api" containerID="cri-o://e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac" gracePeriod=30
Oct 30 20:00:34.783603 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.783251    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:34.838519 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.838482    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-metrics-certs\") pod \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") "
Oct 30 20:00:34.839789 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.839765    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-maas-api-tls\") pod \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") "
Oct 30 20:00:34.840030 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.839998    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-openshift-service-ca\") pod \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") "
Oct 30 20:00:34.840377 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.840354    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-kube-api-access-jcfjv\" (UniqueName: \"kubernetes.io/projected/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-kube-api-access-jcfjv\") pod \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\" (UID: \"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b\") "
Oct 30 20:00:34.855105 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.845223    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-openshift-service-ca" pod "fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" (UID: "fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:34.855105 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.847488    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-metrics-certs" pod "fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" (UID: "fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:34.864342 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.864289    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-maas-api-tls" pod "fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" (UID: "fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:34.865000 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.864960    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-kube-api-access-jcfjv" pod "fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" (UID: "fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b"). InnerVolumeSpecName "kube-api-access-jcfjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:34.995111 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.977751    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-maas-api-tls\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:34.995111 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.977793    6777 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-openshift-service-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:34.995111 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.977810    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcfjv\" (UniqueName: \"kubernetes.io/projected/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-kube-api-access-jcfjv\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:34.995111 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:34.977825    6777 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b-metrics-certs\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:35.046382 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:35.045727    6777 generic.go:356] "Generic (PLEG): container finished" podID="fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" containerID="e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac" exitCode=0
Oct 30 20:00:35.046382 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:35.045782    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb" event={"ID":"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b","Type":"ContainerDied","Data":"e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac"}
Oct 30 20:00:35.046382 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:35.045830    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb" event={"ID":"fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b","Type":"ContainerDied","Data":"16d6d2b36c6215f0ad3c74c06dbea91ce1038868528d187fe625f98e0aa59cf9"}
Oct 30 20:00:35.046382 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:35.045857    6777 scope.go:122] "RemoveContainer" containerID="e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac"
Oct 30 20:00:35.046382 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:35.046000    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"
Oct 30 20:00:35.123772 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:35.123740    6777 scope.go:122] "RemoveContainer" containerID="e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac"
Oct 30 20:00:35.124230 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:00:35.124191    6777 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac\": container with ID starting with e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac not found: ID does not exist" containerID="e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac"
Oct 30 20:00:35.124336 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:35.124245    6777 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac"} err="failed to get container status \"e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac\": rpc error: code = NotFound desc = could not find container \"e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac\": container with ID starting with e8d98b7d97523191b78a278691af8a0c524b910becfda5bd2ef03fddf88717ac not found: ID does not exist"
Oct 30 20:00:35.135245 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:35.135205    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"]
Oct 30 20:00:35.140497 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:35.140464    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l7zwb"]
Oct 30 20:00:36.111583 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.111541    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" path="/var/lib/kubelet/pods/fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b/volumes"
Oct 30 20:00:36.261971 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.261923    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"]
Oct 30 20:00:36.262422 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262405    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" containerName="cleanup"
Oct 30 20:00:36.262478 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262424    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" containerName="cleanup"
Oct 30 20:00:36.262478 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262437    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" containerName="cleanup"
Oct 30 20:00:36.262478 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262442    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" containerName="cleanup"
Oct 30 20:00:36.262478 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262473    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" containerName="cleanup"
Oct 30 20:00:36.262478 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262478    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" containerName="cleanup"
Oct 30 20:00:36.262693 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262486    6777 cpu_manager.go:414] "containerMap: removing container" podUID="fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" containerName="maas-api"
Oct 30 20:00:36.262693 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262490    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" containerName="maas-api"
Oct 30 20:00:36.262693 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262548    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" containerName="cleanup"
Oct 30 20:00:36.262693 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262556    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" containerName="cleanup"
Oct 30 20:00:36.262693 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.262568    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2ffdcc-022d-42b7-9a3c-baeefd2bdb2b" containerName="maas-api"
Oct 30 20:00:36.267081 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.267043    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.273956 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.273933    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:36.274884 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.274860    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 20:00:36.275482 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.275454    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"]
Oct 30 20:00:36.287573 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.287545    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:36.287747 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.287661    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:36.398541 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.398449    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pfqqs\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.398541 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.398491    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkzn\" (UniqueName: \"kubernetes.io/projected/f37416cd-dc9e-417c-82de-3ed996ceba37-kube-api-access-kgkzn\") pod \"maas-api-revoke-sub-f03629db-pfqqs\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.398785 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.398612    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pfqqs\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.499383 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.499333    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pfqqs\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.499566 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.499450    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pfqqs\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.499566 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.499478    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkzn\" (UniqueName: \"kubernetes.io/projected/f37416cd-dc9e-417c-82de-3ed996ceba37-kube-api-access-kgkzn\") pod \"maas-api-revoke-sub-f03629db-pfqqs\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.500156 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.500135    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pfqqs\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.501808 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.501792    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pfqqs\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.512738 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.512718    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkzn\" (UniqueName: \"kubernetes.io/projected/f37416cd-dc9e-417c-82de-3ed996ceba37-kube-api-access-kgkzn\") pod \"maas-api-revoke-sub-f03629db-pfqqs\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.582857 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.582827    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:36.856272 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:36.856248    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"]
Oct 30 20:00:36.858708 ip-10-0-132-19 kubenswrapper[6777]: W1030 20:00:36.858675    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37416cd_dc9e_417c_82de_3ed996ceba37.slice/crio-33b34d810740e2823dde92edd4fd62bbeca24997cc27d64eccc2ae44dc383812 WatchSource:0}: Error finding container 33b34d810740e2823dde92edd4fd62bbeca24997cc27d64eccc2ae44dc383812: Status 404 returned error can't find the container with id 33b34d810740e2823dde92edd4fd62bbeca24997cc27d64eccc2ae44dc383812
Oct 30 20:00:37.025203 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:37.025172    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"]
Oct 30 20:00:37.028048 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:37.028020    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vd2bc"]
Oct 30 20:00:37.057719 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:37.057693    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs" event={"ID":"f37416cd-dc9e-417c-82de-3ed996ceba37","Type":"ContainerStarted","Data":"33b34d810740e2823dde92edd4fd62bbeca24997cc27d64eccc2ae44dc383812"}
Oct 30 20:00:38.064564 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:38.064534    6777 generic.go:356] "Generic (PLEG): container finished" podID="f37416cd-dc9e-417c-82de-3ed996ceba37" containerID="0c0493c4f883c7c5f886a5f918c76e2c3227fdf8e9a3899fde57bdd6bdb27623" exitCode=0
Oct 30 20:00:38.064959 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:38.064614    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs" event={"ID":"f37416cd-dc9e-417c-82de-3ed996ceba37","Type":"ContainerDied","Data":"0c0493c4f883c7c5f886a5f918c76e2c3227fdf8e9a3899fde57bdd6bdb27623"}
Oct 30 20:00:38.110770 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:38.110741    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cc9725-177a-4ba3-9dbb-ca262b3e8ef8" path="/var/lib/kubelet/pods/93cc9725-177a-4ba3-9dbb-ca262b3e8ef8/volumes"
Oct 30 20:00:39.201848 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.201822    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:39.325168 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.325077    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-xks-ca\") pod \"f37416cd-dc9e-417c-82de-3ed996ceba37\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") "
Oct 30 20:00:39.325331 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.325188    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-openshift-ca\") pod \"f37416cd-dc9e-417c-82de-3ed996ceba37\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") "
Oct 30 20:00:39.325331 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.325323    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f37416cd-dc9e-417c-82de-3ed996ceba37-kube-api-access-kgkzn\" (UniqueName: \"kubernetes.io/projected/f37416cd-dc9e-417c-82de-3ed996ceba37-kube-api-access-kgkzn\") pod \"f37416cd-dc9e-417c-82de-3ed996ceba37\" (UID: \"f37416cd-dc9e-417c-82de-3ed996ceba37\") "
Oct 30 20:00:39.325492 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.325464    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-openshift-ca" pod "f37416cd-dc9e-417c-82de-3ed996ceba37" (UID: "f37416cd-dc9e-417c-82de-3ed996ceba37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:39.325682 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.325666    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.327212 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.327182    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-xks-ca" pod "f37416cd-dc9e-417c-82de-3ed996ceba37" (UID: "f37416cd-dc9e-417c-82de-3ed996ceba37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:39.327288 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.327246    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37416cd-dc9e-417c-82de-3ed996ceba37-kube-api-access-kgkzn" pod "f37416cd-dc9e-417c-82de-3ed996ceba37" (UID: "f37416cd-dc9e-417c-82de-3ed996ceba37"). InnerVolumeSpecName "kube-api-access-kgkzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:39.426854 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.426819    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37416cd-dc9e-417c-82de-3ed996ceba37-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.426854 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:39.426846    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgkzn\" (UniqueName: \"kubernetes.io/projected/f37416cd-dc9e-417c-82de-3ed996ceba37-kube-api-access-kgkzn\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:40.076255 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:40.076209    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs" event={"ID":"f37416cd-dc9e-417c-82de-3ed996ceba37","Type":"ContainerDied","Data":"33b34d810740e2823dde92edd4fd62bbeca24997cc27d64eccc2ae44dc383812"}
Oct 30 20:00:40.076255 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:40.076248    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pfqqs"
Oct 30 20:00:40.076476 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:40.076268    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b34d810740e2823dde92edd4fd62bbeca24997cc27d64eccc2ae44dc383812"
Oct 30 20:00:43.637717 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:00:43.637674    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 20:00:46.253985 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:46.253950    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 20:00:51.308141 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.308105    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"]
Oct 30 20:00:51.308779 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.308756    6777 cpu_manager.go:414] "containerMap: removing container" podUID="f37416cd-dc9e-417c-82de-3ed996ceba37" containerName="revoke-keys"
Oct 30 20:00:51.308871 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.308781    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f37416cd-dc9e-417c-82de-3ed996ceba37" containerName="revoke-keys"
Oct 30 20:00:51.308931 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.308909    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="f37416cd-dc9e-417c-82de-3ed996ceba37" containerName="revoke-keys"
Oct 30 20:00:51.308985 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.308929    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6201bdc0-f971-426d-bb81-d3bde264db96" containerName="cleanup"
Oct 30 20:00:51.312334 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.312307    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.319833 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.319764    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"]
Oct 30 20:00:51.320875 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.320484    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:51.320875 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.320603    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 20:00:51.320875 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.320642    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:51.321223 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.321206    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:51.464735 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.464701    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-k5k95\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.464913 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.464754    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcdj\" (UniqueName: \"kubernetes.io/projected/c084edff-089a-4bb8-9842-4d5c170f1d23-kube-api-access-nqcdj\") pod \"maas-api-revoke-sub-0165d7d2-k5k95\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.464913 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.464866    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-k5k95\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.565596 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.565524    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcdj\" (UniqueName: \"kubernetes.io/projected/c084edff-089a-4bb8-9842-4d5c170f1d23-kube-api-access-nqcdj\") pod \"maas-api-revoke-sub-0165d7d2-k5k95\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.565731 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.565608    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-k5k95\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.565731 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.565640    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-k5k95\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.566232 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.566210    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-k5k95\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.567917 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.567898    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-k5k95\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.578462 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.578442    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcdj\" (UniqueName: \"kubernetes.io/projected/c084edff-089a-4bb8-9842-4d5c170f1d23-kube-api-access-nqcdj\") pod \"maas-api-revoke-sub-0165d7d2-k5k95\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.627375 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.627354    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:51.902347 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:51.902318    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"]
Oct 30 20:00:51.904280 ip-10-0-132-19 kubenswrapper[6777]: W1030 20:00:51.904254    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc084edff_089a_4bb8_9842_4d5c170f1d23.slice/crio-afc8368df96ef8b0af3806804c3bce0d61d35e30c13f98573051366dc9a2e791 WatchSource:0}: Error finding container afc8368df96ef8b0af3806804c3bce0d61d35e30c13f98573051366dc9a2e791: Status 404 returned error can't find the container with id afc8368df96ef8b0af3806804c3bce0d61d35e30c13f98573051366dc9a2e791
Oct 30 20:00:52.156222 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:52.156018    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95" event={"ID":"c084edff-089a-4bb8-9842-4d5c170f1d23","Type":"ContainerStarted","Data":"afc8368df96ef8b0af3806804c3bce0d61d35e30c13f98573051366dc9a2e791"}
Oct 30 20:00:53.163025 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:53.162976    6777 generic.go:356] "Generic (PLEG): container finished" podID="c084edff-089a-4bb8-9842-4d5c170f1d23" containerID="170009d26653243203844d8b4e492d1b43a3d932e816a675ab4c74fcc7a9ea98" exitCode=0
Oct 30 20:00:53.163400 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:53.163053    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95" event={"ID":"c084edff-089a-4bb8-9842-4d5c170f1d23","Type":"ContainerDied","Data":"170009d26653243203844d8b4e492d1b43a3d932e816a675ab4c74fcc7a9ea98"}
Oct 30 20:00:54.309952 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.309920    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:54.492501 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.492420    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c084edff-089a-4bb8-9842-4d5c170f1d23-kube-api-access-nqcdj\" (UniqueName: \"kubernetes.io/projected/c084edff-089a-4bb8-9842-4d5c170f1d23-kube-api-access-nqcdj\") pod \"c084edff-089a-4bb8-9842-4d5c170f1d23\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") "
Oct 30 20:00:54.492501 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.492468    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-xks-ca\") pod \"c084edff-089a-4bb8-9842-4d5c170f1d23\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") "
Oct 30 20:00:54.492501 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.492497    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-openshift-ca\") pod \"c084edff-089a-4bb8-9842-4d5c170f1d23\" (UID: \"c084edff-089a-4bb8-9842-4d5c170f1d23\") "
Oct 30 20:00:54.492906 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.492884    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-openshift-ca" pod "c084edff-089a-4bb8-9842-4d5c170f1d23" (UID: "c084edff-089a-4bb8-9842-4d5c170f1d23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:54.494479 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.494457    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c084edff-089a-4bb8-9842-4d5c170f1d23-kube-api-access-nqcdj" pod "c084edff-089a-4bb8-9842-4d5c170f1d23" (UID: "c084edff-089a-4bb8-9842-4d5c170f1d23"). InnerVolumeSpecName "kube-api-access-nqcdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:54.494583 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.494503    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-xks-ca" pod "c084edff-089a-4bb8-9842-4d5c170f1d23" (UID: "c084edff-089a-4bb8-9842-4d5c170f1d23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:54.594255 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.594218    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqcdj\" (UniqueName: \"kubernetes.io/projected/c084edff-089a-4bb8-9842-4d5c170f1d23-kube-api-access-nqcdj\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:54.594255 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.594253    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:54.594255 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:54.594263    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c084edff-089a-4bb8-9842-4d5c170f1d23-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:55.174854 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:55.174824    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95"
Oct 30 20:00:55.174854 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:55.174841    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k5k95" event={"ID":"c084edff-089a-4bb8-9842-4d5c170f1d23","Type":"ContainerDied","Data":"afc8368df96ef8b0af3806804c3bce0d61d35e30c13f98573051366dc9a2e791"}
Oct 30 20:00:55.175131 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:00:55.174888    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc8368df96ef8b0af3806804c3bce0d61d35e30c13f98573051366dc9a2e791"
Oct 30 20:01:02.025183 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.025153    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"]
Oct 30 20:01:02.025598 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.025586    6777 cpu_manager.go:414] "containerMap: removing container" podUID="c084edff-089a-4bb8-9842-4d5c170f1d23" containerName="revoke-keys"
Oct 30 20:01:02.025646 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.025599    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c084edff-089a-4bb8-9842-4d5c170f1d23" containerName="revoke-keys"
Oct 30 20:01:02.025688 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.025670    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="c084edff-089a-4bb8-9842-4d5c170f1d23" containerName="revoke-keys"
Oct 30 20:01:02.028787 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.028763    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.035856 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.035830    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:02.036033 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.035899    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 20:01:02.036181 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.035945    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:02.036181 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.036027    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:02.040097 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.039071    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"]
Oct 30 20:01:02.065125 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.065096    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv9cv\" (UniqueName: \"kubernetes.io/projected/6eb28b25-ed75-438a-84ee-f8bf20027056-kube-api-access-dv9cv\") pod \"maas-api-revoke-sub-6b19338f-9p8mm\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.065284 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.065132    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9p8mm\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.065284 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.065160    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9p8mm\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.166025 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.165993    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv9cv\" (UniqueName: \"kubernetes.io/projected/6eb28b25-ed75-438a-84ee-f8bf20027056-kube-api-access-dv9cv\") pod \"maas-api-revoke-sub-6b19338f-9p8mm\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.166201 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.166027    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9p8mm\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.166201 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.166095    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9p8mm\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.166784 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.166761    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9p8mm\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.168350 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.168324    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9p8mm\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.178922 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.178902    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv9cv\" (UniqueName: \"kubernetes.io/projected/6eb28b25-ed75-438a-84ee-f8bf20027056-kube-api-access-dv9cv\") pod \"maas-api-revoke-sub-6b19338f-9p8mm\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.346183 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.346154    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:02.642826 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:02.637302    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"]
Oct 30 20:01:02.650226 ip-10-0-132-19 kubenswrapper[6777]: W1030 20:01:02.649802    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb28b25_ed75_438a_84ee_f8bf20027056.slice/crio-ce3e060eeae2488c6e69b25a6a537b17ab0d981ae158c62c709bbd8017443924 WatchSource:0}: Error finding container ce3e060eeae2488c6e69b25a6a537b17ab0d981ae158c62c709bbd8017443924: Status 404 returned error can't find the container with id ce3e060eeae2488c6e69b25a6a537b17ab0d981ae158c62c709bbd8017443924
Oct 30 20:01:03.217717 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:03.217625    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm" event={"ID":"6eb28b25-ed75-438a-84ee-f8bf20027056","Type":"ContainerStarted","Data":"a6c43fc7c326b1248c980ecdca37b7cf94f62187494fe54ad3ecafb9648a5ec0"}
Oct 30 20:01:03.217717 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:03.217678    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm" event={"ID":"6eb28b25-ed75-438a-84ee-f8bf20027056","Type":"ContainerStarted","Data":"ce3e060eeae2488c6e69b25a6a537b17ab0d981ae158c62c709bbd8017443924"}
Oct 30 20:01:03.237875 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:03.237825    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm" podStartSLOduration=1.237812217 podStartE2EDuration="1.237812217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:03.236329399 +0000 UTC m=+3245.233244788" watchObservedRunningTime="2026-10-30 20:01:03.237812217 +0000 UTC m=+3245.234727599"
Oct 30 20:01:04.223861 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:04.223826    6777 generic.go:356] "Generic (PLEG): container finished" podID="6eb28b25-ed75-438a-84ee-f8bf20027056" containerID="a6c43fc7c326b1248c980ecdca37b7cf94f62187494fe54ad3ecafb9648a5ec0" exitCode=0
Oct 30 20:01:04.224264 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:04.223912    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm" event={"ID":"6eb28b25-ed75-438a-84ee-f8bf20027056","Type":"ContainerDied","Data":"a6c43fc7c326b1248c980ecdca37b7cf94f62187494fe54ad3ecafb9648a5ec0"}
Oct 30 20:01:05.357233 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.357207    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:05.398750 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.398716    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-openshift-ca\") pod \"6eb28b25-ed75-438a-84ee-f8bf20027056\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") "
Oct 30 20:01:05.398943 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.398766    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-xks-ca\") pod \"6eb28b25-ed75-438a-84ee-f8bf20027056\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") "
Oct 30 20:01:05.398943 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.398812    6777 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb28b25-ed75-438a-84ee-f8bf20027056-kube-api-access-dv9cv\" (UniqueName: \"kubernetes.io/projected/6eb28b25-ed75-438a-84ee-f8bf20027056-kube-api-access-dv9cv\") pod \"6eb28b25-ed75-438a-84ee-f8bf20027056\" (UID: \"6eb28b25-ed75-438a-84ee-f8bf20027056\") "
Oct 30 20:01:05.399147 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.399118    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-openshift-ca" pod "6eb28b25-ed75-438a-84ee-f8bf20027056" (UID: "6eb28b25-ed75-438a-84ee-f8bf20027056"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:05.400860 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.400835    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-xks-ca" pod "6eb28b25-ed75-438a-84ee-f8bf20027056" (UID: "6eb28b25-ed75-438a-84ee-f8bf20027056"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:05.400968 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.400917    6777 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb28b25-ed75-438a-84ee-f8bf20027056-kube-api-access-dv9cv" pod "6eb28b25-ed75-438a-84ee-f8bf20027056" (UID: "6eb28b25-ed75-438a-84ee-f8bf20027056"). InnerVolumeSpecName "kube-api-access-dv9cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:05.500358 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.500281    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-openshift-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:05.500358 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.500309    6777 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb28b25-ed75-438a-84ee-f8bf20027056-maas-api-xks-ca\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:05.500358 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:05.500319    6777 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv9cv\" (UniqueName: \"kubernetes.io/projected/6eb28b25-ed75-438a-84ee-f8bf20027056-kube-api-access-dv9cv\") on node \"ip-10-0-132-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:06.234770 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:06.234742    6777 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm"
Oct 30 20:01:06.234770 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:06.234748    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9p8mm" event={"ID":"6eb28b25-ed75-438a-84ee-f8bf20027056","Type":"ContainerDied","Data":"ce3e060eeae2488c6e69b25a6a537b17ab0d981ae158c62c709bbd8017443924"}
Oct 30 20:01:06.234957 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:06.234791    6777 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce3e060eeae2488c6e69b25a6a537b17ab0d981ae158c62c709bbd8017443924"
Oct 30 20:01:06.932476 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:06.932447    6777 scope.go:122] "RemoveContainer" containerID="0121b97ccead723b2195bb833828f3b4678b01ea23837871247b03d255c045f2"
Oct 30 20:01:06.941564 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:06.941549    6777 scope.go:122] "RemoveContainer" containerID="ca7f346cf226fb2edb8713316a12e62ef54b2e80d35360ba0f7f854c233779f4"
Oct 30 20:01:12.788237 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:12.788156    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lzfhp"]
Oct 30 20:01:13.412297 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:13.412265    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f58996d-sl6pb_043e9e20-e8e8-4403-a5d9-85cfd385d5cd/authorino/0.log"
Oct 30 20:01:13.739144 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:01:13.737278    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 20:01:26.030595 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:26.030567    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c94c66984-4t9lk_35a6edc2-ad9d-4077-b69d-e67245ba689d/copy-manifests/0.log"
Oct 30 20:01:26.098198 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:26.098169    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c94c66984-4t9lk_35a6edc2-ad9d-4077-b69d-e67245ba689d/manager/0.log"
Oct 30 20:01:27.897891 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:27.897860    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-k5k95_c084edff-089a-4bb8-9842-4d5c170f1d23/revoke-keys/0.log"
Oct 30 20:01:28.087478 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:28.087451    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-288vz_896971d6-9846-492a-b434-08ee4cf45b32/revoke-keys/0.log"
Oct 30 20:01:28.273175 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:28.273098    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-nb5qd_92654488-bdba-40b5-b9a7-4dd95658f441/revoke-keys/0.log"
Oct 30 20:01:28.467848 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:28.467824    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9p8mm_6eb28b25-ed75-438a-84ee-f8bf20027056/revoke-keys/0.log"
Oct 30 20:01:28.652733 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:28.652709    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-98bf6ba1-plhjg_0a69ba63-c9b6-4b8e-922f-3833098e4dfc/revoke-keys/0.log"
Oct 30 20:01:29.007927 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:29.007851    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-lbrqn_2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a/revoke-keys/0.log"
Oct 30 20:01:29.188957 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:29.188930    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-86smx_66012094-5a47-4234-8c38-13dcd59901ec/revoke-keys/0.log"
Oct 30 20:01:29.366836 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:29.366801    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-pfqqs_f37416cd-dc9e-417c-82de-3ed996ceba37/revoke-keys/0.log"
Oct 30 20:01:31.091054 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:31.091023    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f58996d-sl6pb_043e9e20-e8e8-4403-a5d9-85cfd385d5cd/authorino/0.log"
Oct 30 20:01:32.144153 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:32.144122    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lzfhp_77d9131c-32d0-4962-9e0c-dbbcaa8a275a/limitador/0.log"
Oct 30 20:01:32.320976 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:32.320944    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-d7p7h_cdf5be3b-460b-4e99-afed-5f4fc0b287cb/manager/0.log"
Oct 30 20:01:33.483277 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:33.483243    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-678bf485d7-bnc5s_5e038566-ad43-4e4b-b314-ade25d7b389b/kube-auth-proxy/0.log"
Oct 30 20:01:34.619278 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:34.619246    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-m5mbs_ae6b2aed-dbe2-4934-8c89-00f2f35ce325/istio-proxy/0.log"
Oct 30 20:01:34.804958 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:34.804923    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-pthrg_8084f1f2-0fb7-4dbb-b255-4ffcef554e9c/istio-proxy/0.log"
Oct 30 20:01:35.367096 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:35.367043    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wz7wl_c5a9e30d-41c9-45a4-a6ce-396e5d3f8952/istio-proxy/0.log"
Oct 30 20:01:35.548559 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:35.548526    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-xlxqw_e579180c-22b8-4904-b2a3-acad28497f49/istio-proxy/0.log"
Oct 30 20:01:35.724142 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:35.724050    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-svc7x_d5d7f713-e99d-47f0-a335-759341ccd335/payload-pre-processing/0.log"
Oct 30 20:01:35.915288 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:35.915258    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 20:01:37.280287 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:37.280252    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mlz74_0c45319f-1d63-435a-85b6-5e49094db5be/main/0.log"
Oct 30 20:01:38.001836 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:38.001809    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf_d3d7cb45-4e6d-466b-aa7b-498e2a682c99/main/0.log"
Oct 30 20:01:38.183659 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:38.183627    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28_d9cbeb26-fde8-410b-b600-b3d536fed7c8/main/0.log"
Oct 30 20:01:39.079101 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:39.079047    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f58996d-sl6pb_043e9e20-e8e8-4403-a5d9-85cfd385d5cd/authorino/0.log"
Oct 30 20:01:43.638769 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:01:43.638729    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 20:01:50.110700 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:50.110668    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c94c66984-4t9lk_35a6edc2-ad9d-4077-b69d-e67245ba689d/manager/0.log"
Oct 30 20:01:50.155910 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:50.155886    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c94c66984-4t9lk_35a6edc2-ad9d-4077-b69d-e67245ba689d/copy-manifests/0.log"
Oct 30 20:01:51.720644 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:51.720616    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-k5k95_c084edff-089a-4bb8-9842-4d5c170f1d23/revoke-keys/0.log"
Oct 30 20:01:51.903450 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:51.903422    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-288vz_896971d6-9846-492a-b434-08ee4cf45b32/revoke-keys/0.log"
Oct 30 20:01:52.087484 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:52.087457    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-nb5qd_92654488-bdba-40b5-b9a7-4dd95658f441/revoke-keys/0.log"
Oct 30 20:01:52.274220 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:52.274192    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9p8mm_6eb28b25-ed75-438a-84ee-f8bf20027056/revoke-keys/0.log"
Oct 30 20:01:52.470328 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:52.470241    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-98bf6ba1-plhjg_0a69ba63-c9b6-4b8e-922f-3833098e4dfc/revoke-keys/0.log"
Oct 30 20:01:52.835177 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:52.835148    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-lbrqn_2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a/revoke-keys/0.log"
Oct 30 20:01:53.011811 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:53.011777    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-86smx_66012094-5a47-4234-8c38-13dcd59901ec/revoke-keys/0.log"
Oct 30 20:01:53.197839 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:53.197753    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-pfqqs_f37416cd-dc9e-417c-82de-3ed996ceba37/revoke-keys/0.log"
Oct 30 20:01:54.505944 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:54.505906    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f58996d-sl6pb_043e9e20-e8e8-4403-a5d9-85cfd385d5cd/authorino/0.log"
Oct 30 20:01:55.551142 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:55.551113    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lzfhp_77d9131c-32d0-4962-9e0c-dbbcaa8a275a/limitador/0.log"
Oct 30 20:01:55.740673 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:55.740642    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-d7p7h_cdf5be3b-460b-4e99-afed-5f4fc0b287cb/manager/0.log"
Oct 30 20:01:56.689334 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:56.689303    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-678bf485d7-bnc5s_5e038566-ad43-4e4b-b314-ade25d7b389b/kube-auth-proxy/0.log"
Oct 30 20:01:57.833537 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:57.833497    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-m5mbs_ae6b2aed-dbe2-4934-8c89-00f2f35ce325/istio-proxy/0.log"
Oct 30 20:01:58.018375 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:58.018327    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-pthrg_8084f1f2-0fb7-4dbb-b255-4ffcef554e9c/istio-proxy/0.log"
Oct 30 20:01:58.584853 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:58.584825    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wz7wl_c5a9e30d-41c9-45a4-a6ce-396e5d3f8952/istio-proxy/0.log"
Oct 30 20:01:58.764379 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:58.764329    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-xlxqw_e579180c-22b8-4904-b2a3-acad28497f49/istio-proxy/0.log"
Oct 30 20:01:58.938220 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:58.938136    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-svc7x_d5d7f713-e99d-47f0-a335-759341ccd335/payload-pre-processing/0.log"
Oct 30 20:01:59.107416 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:01:59.107387    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 20:02:00.034358 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:00.034320    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mlz74_0c45319f-1d63-435a-85b6-5e49094db5be/main/0.log"
Oct 30 20:02:00.765518 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:00.765490    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-l8fjf_d3d7cb45-4e6d-466b-aa7b-498e2a682c99/main/0.log"
Oct 30 20:02:00.955630 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:00.955606    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-kvv28_d9cbeb26-fde8-410b-b600-b3d536fed7c8/main/0.log"
Oct 30 20:02:06.979677 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:06.979647    6777 scope.go:122] "RemoveContainer" containerID="60c08de166ee5266d3a8374581f0148478e00899becd758887e27b8fed9d1811"
Oct 30 20:02:06.988021 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:06.987994    6777 scope.go:122] "RemoveContainer" containerID="915f59d9266340a0b8c6ee423e8d49fa9d9c81d21140e1b73bf6d8d92f9dedf9"
Oct 30 20:02:13.641288 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:02:13.641247    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 20:02:21.025882 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:21.025841    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"]
Oct 30 20:02:21.029772 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:21.029737    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-86smx"]
Oct 30 20:02:22.120717 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:22.120673    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66012094-5a47-4234-8c38-13dcd59901ec" path="/var/lib/kubelet/pods/66012094-5a47-4234-8c38-13dcd59901ec/volumes"
Oct 30 20:02:37.028367 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:37.028321    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"]
Oct 30 20:02:37.030414 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:37.030386    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lbrqn"]
Oct 30 20:02:38.111172 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:38.111137    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a" path="/var/lib/kubelet/pods/2abbb2ee-29fd-4f1c-af8b-cd2b72cf7c4a/volumes"
Oct 30 20:02:39.133842 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:39.133762    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 20:02:39.328401 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:02:39.328372    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 20:02:43.641103 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:02:43.641049    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 20:03:13.640403 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:03:13.640362    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 20:03:24.735792 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:24.735760    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 20:03:24.903359 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:24.903326    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 20:03:25.221948 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.221903    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trmds/must-gather-b7lk8"]
Oct 30 20:03:25.222498 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.222470    6777 cpu_manager.go:414] "containerMap: removing container" podUID="6eb28b25-ed75-438a-84ee-f8bf20027056" containerName="revoke-keys"
Oct 30 20:03:25.222498 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.222496    6777 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb28b25-ed75-438a-84ee-f8bf20027056" containerName="revoke-keys"
Oct 30 20:03:25.222676 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.222653    6777 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb28b25-ed75-438a-84ee-f8bf20027056" containerName="revoke-keys"
Oct 30 20:03:25.225819 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.225801    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmds/must-gather-b7lk8"
Oct 30 20:03:25.231153 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.231127    6777 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-trmds\"/\"default-dockercfg-mnw2j\""
Oct 30 20:03:25.231283 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.231191    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmds\"/\"kube-root-ca.crt\""
Oct 30 20:03:25.231283 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.231191    6777 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmds\"/\"openshift-service-ca.crt\""
Oct 30 20:03:25.250980 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.250954    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmds/must-gather-b7lk8"]
Oct 30 20:03:25.284326 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.284288    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grn44\" (UniqueName: \"kubernetes.io/projected/5e3bcd9b-f612-4c8b-a745-8ec77fb38480-kube-api-access-grn44\") pod \"must-gather-b7lk8\" (UID: \"5e3bcd9b-f612-4c8b-a745-8ec77fb38480\") " pod="openshift-must-gather-trmds/must-gather-b7lk8"
Oct 30 20:03:25.284504 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.284445    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5e3bcd9b-f612-4c8b-a745-8ec77fb38480-must-gather-output\") pod \"must-gather-b7lk8\" (UID: \"5e3bcd9b-f612-4c8b-a745-8ec77fb38480\") " pod="openshift-must-gather-trmds/must-gather-b7lk8"
Oct 30 20:03:25.385618 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.385579    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grn44\" (UniqueName: \"kubernetes.io/projected/5e3bcd9b-f612-4c8b-a745-8ec77fb38480-kube-api-access-grn44\") pod \"must-gather-b7lk8\" (UID: \"5e3bcd9b-f612-4c8b-a745-8ec77fb38480\") " pod="openshift-must-gather-trmds/must-gather-b7lk8"
Oct 30 20:03:25.385822 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.385678    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5e3bcd9b-f612-4c8b-a745-8ec77fb38480-must-gather-output\") pod \"must-gather-b7lk8\" (UID: \"5e3bcd9b-f612-4c8b-a745-8ec77fb38480\") " pod="openshift-must-gather-trmds/must-gather-b7lk8"
Oct 30 20:03:25.386000 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.385983    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5e3bcd9b-f612-4c8b-a745-8ec77fb38480-must-gather-output\") pod \"must-gather-b7lk8\" (UID: \"5e3bcd9b-f612-4c8b-a745-8ec77fb38480\") " pod="openshift-must-gather-trmds/must-gather-b7lk8"
Oct 30 20:03:25.398301 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.398267    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grn44\" (UniqueName: \"kubernetes.io/projected/5e3bcd9b-f612-4c8b-a745-8ec77fb38480-kube-api-access-grn44\") pod \"must-gather-b7lk8\" (UID: \"5e3bcd9b-f612-4c8b-a745-8ec77fb38480\") " pod="openshift-must-gather-trmds/must-gather-b7lk8"
Oct 30 20:03:25.540090 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.540054    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmds/must-gather-b7lk8"
Oct 30 20:03:25.812916 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.812875    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmds/must-gather-b7lk8"]
Oct 30 20:03:25.815151 ip-10-0-132-19 kubenswrapper[6777]: W1030 20:03:25.815123    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e3bcd9b_f612_4c8b_a745_8ec77fb38480.slice/crio-aa33e29f43373edae4985ecdcad227991ff5db4c0306a7ba5c7ececa14d435b6 WatchSource:0}: Error finding container aa33e29f43373edae4985ecdcad227991ff5db4c0306a7ba5c7ececa14d435b6: Status 404 returned error can't find the container with id aa33e29f43373edae4985ecdcad227991ff5db4c0306a7ba5c7ececa14d435b6
Oct 30 20:03:25.974128 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:25.974094    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmds/must-gather-b7lk8" event={"ID":"5e3bcd9b-f612-4c8b-a745-8ec77fb38480","Type":"ContainerStarted","Data":"aa33e29f43373edae4985ecdcad227991ff5db4c0306a7ba5c7ececa14d435b6"}
Oct 30 20:03:26.985039 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:26.984943    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmds/must-gather-b7lk8" event={"ID":"5e3bcd9b-f612-4c8b-a745-8ec77fb38480","Type":"ContainerStarted","Data":"53d5dc2e3f0e86ca8ad853f8395a6edda62ea2a91258ed1b3f32fd3a4e73fb20"}
Oct 30 20:03:27.026801 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:27.026770    6777 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"]
Oct 30 20:03:27.029411 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:27.029384    6777 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf6ba1-plhjg"]
Oct 30 20:03:27.994360 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:27.993390    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmds/must-gather-b7lk8" event={"ID":"5e3bcd9b-f612-4c8b-a745-8ec77fb38480","Type":"ContainerStarted","Data":"3ae07c59b982b3795645f13530f29f8e4a2559625710af46294cabe73b1e6ba4"}
Oct 30 20:03:28.017309 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:28.017243    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-trmds/must-gather-b7lk8" podStartSLOduration=2.123667819 podStartE2EDuration="3.01722484s" totalImagesPullingTime="893.557021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:28.017226272 +0000 UTC m=+3390.014141659" watchObservedRunningTime="2026-10-30 20:03:28.01722484 +0000 UTC m=+3390.014140226"
Oct 30 20:03:28.112703 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:28.112672    6777 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a69ba63-c9b6-4b8e-922f-3833098e4dfc" path="/var/lib/kubelet/pods/0a69ba63-c9b6-4b8e-922f-3833098e4dfc/volumes"
Oct 30 20:03:28.989992 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:28.989958    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tvwkt_baec05dd-6d69-41d9-ad73-4179b17ba8f4/global-pull-secret-syncer/0.log"
Oct 30 20:03:29.229791 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:29.229754    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-24ljs_1b26e19e-b8e9-496a-bfcb-6cd5add61e6e/konnectivity-agent/0.log"
Oct 30 20:03:29.457190 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:29.457161    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-19.us-east-2.compute.internal_b20bfab463977c229c9db20c57c04c0b/haproxy/0.log"
Oct 30 20:03:36.433818 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:36.433772    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f58996d-sl6pb_043e9e20-e8e8-4403-a5d9-85cfd385d5cd/authorino/0.log"
Oct 30 20:03:36.677078 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:36.666029    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lzfhp_77d9131c-32d0-4962-9e0c-dbbcaa8a275a/limitador/0.log"
Oct 30 20:03:36.755869 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:36.755782    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-d7p7h_cdf5be3b-460b-4e99-afed-5f4fc0b287cb/manager/0.log"
Oct 30 20:03:38.104853 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.104823    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_73033c1c-bdf2-41a5-8a23-8fe025e3df89/alertmanager/0.log"
Oct 30 20:03:38.143169 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.142729    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_73033c1c-bdf2-41a5-8a23-8fe025e3df89/config-reloader/0.log"
Oct 30 20:03:38.179905 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.179873    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_73033c1c-bdf2-41a5-8a23-8fe025e3df89/kube-rbac-proxy-web/0.log"
Oct 30 20:03:38.212352 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.212326    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_73033c1c-bdf2-41a5-8a23-8fe025e3df89/kube-rbac-proxy/0.log"
Oct 30 20:03:38.248561 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.248533    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_73033c1c-bdf2-41a5-8a23-8fe025e3df89/kube-rbac-proxy-metric/0.log"
Oct 30 20:03:38.285331 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.285297    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_73033c1c-bdf2-41a5-8a23-8fe025e3df89/prom-label-proxy/0.log"
Oct 30 20:03:38.317196 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.317168    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_73033c1c-bdf2-41a5-8a23-8fe025e3df89/init-config-reloader/0.log"
Oct 30 20:03:38.419033 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.418921    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-976bs_558fbbeb-c828-4767-aecc-8c319ab91b35/kube-state-metrics/0.log"
Oct 30 20:03:38.443515 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.443480    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-976bs_558fbbeb-c828-4767-aecc-8c319ab91b35/kube-rbac-proxy-main/0.log"
Oct 30 20:03:38.476159 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.476128    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-976bs_558fbbeb-c828-4767-aecc-8c319ab91b35/kube-rbac-proxy-self/0.log"
Oct 30 20:03:38.514078 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.514024    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-665cc87487-rzd5l_f3925cae-4e22-41e0-b510-7c5cec2bfd17/metrics-server/0.log"
Oct 30 20:03:38.551348 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.551274    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-6mh9p_77ff1252-ac70-4fad-a824-99f3c1eb860e/monitoring-plugin/0.log"
Oct 30 20:03:38.786835 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.786727    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tsr7c_91513898-3858-4991-afad-cb5999ab98d0/node-exporter/0.log"
Oct 30 20:03:38.821300 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.821270    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tsr7c_91513898-3858-4991-afad-cb5999ab98d0/kube-rbac-proxy/0.log"
Oct 30 20:03:38.853838 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.853810    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tsr7c_91513898-3858-4991-afad-cb5999ab98d0/init-textfile/0.log"
Oct 30 20:03:38.895076 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.891565    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6lpsn_61317c8f-5a2e-4268-b085-df795ec4baa6/kube-rbac-proxy-main/0.log"
Oct 30 20:03:38.924260 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.924190    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6lpsn_61317c8f-5a2e-4268-b085-df795ec4baa6/kube-rbac-proxy-self/0.log"
Oct 30 20:03:38.962800 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:38.962741    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6lpsn_61317c8f-5a2e-4268-b085-df795ec4baa6/openshift-state-metrics/0.log"
Oct 30 20:03:39.014781 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.014748    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_586bce6a-ff5d-4531-935d-6cada4099b7d/prometheus/0.log"
Oct 30 20:03:39.045515 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.045438    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_586bce6a-ff5d-4531-935d-6cada4099b7d/config-reloader/0.log"
Oct 30 20:03:39.077126 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.077092    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_586bce6a-ff5d-4531-935d-6cada4099b7d/thanos-sidecar/0.log"
Oct 30 20:03:39.107827 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.107791    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_586bce6a-ff5d-4531-935d-6cada4099b7d/kube-rbac-proxy-web/0.log"
Oct 30 20:03:39.139449 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.139417    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_586bce6a-ff5d-4531-935d-6cada4099b7d/kube-rbac-proxy/0.log"
Oct 30 20:03:39.175306 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.175272    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_586bce6a-ff5d-4531-935d-6cada4099b7d/kube-rbac-proxy-thanos/0.log"
Oct 30 20:03:39.218516 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.218481    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_586bce6a-ff5d-4531-935d-6cada4099b7d/init-config-reloader/0.log"
Oct 30 20:03:39.399672 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.399581    6777 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"]
Oct 30 20:03:39.404933 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.404904    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.413443 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.413406    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"]
Oct 30 20:03:39.481398 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.481366    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-podres\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.481598 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.481411    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-sys\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.481598 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.481507    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-proc\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.481707 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.481645    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-lib-modules\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.481754 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.481714    6777 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlzw\" (UniqueName: \"kubernetes.io/projected/9aea2aed-c550-4784-b822-332be0ae850e-kube-api-access-pqlzw\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.493129 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.493092    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77856dc4d7-j9gps_d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/thanos-query/0.log"
Oct 30 20:03:39.530789 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.530749    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77856dc4d7-j9gps_d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/kube-rbac-proxy-web/0.log"
Oct 30 20:03:39.561903 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.561857    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77856dc4d7-j9gps_d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/kube-rbac-proxy/0.log"
Oct 30 20:03:39.582476 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.582443    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-lib-modules\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.582700 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.582499    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlzw\" (UniqueName: \"kubernetes.io/projected/9aea2aed-c550-4784-b822-332be0ae850e-kube-api-access-pqlzw\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.582700 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.582558    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-podres\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.582700 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.582580    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-sys\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.582700 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.582597    6777 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-proc\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.582700 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.582617    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-lib-modules\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.582700 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.582674    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-proc\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.582700 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.582680    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-podres\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.582700 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.582680    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9aea2aed-c550-4784-b822-332be0ae850e-sys\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.601090 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.599122    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77856dc4d7-j9gps_d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/prom-label-proxy/0.log"
Oct 30 20:03:39.603977 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.603946    6777 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlzw\" (UniqueName: \"kubernetes.io/projected/9aea2aed-c550-4784-b822-332be0ae850e-kube-api-access-pqlzw\") pod \"perf-node-gather-daemonset-7vlzz\" (UID: \"9aea2aed-c550-4784-b822-332be0ae850e\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:39.632311 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.632282    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77856dc4d7-j9gps_d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/kube-rbac-proxy-rules/0.log"
Oct 30 20:03:39.666433 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.666334    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77856dc4d7-j9gps_d5e0ef42-c5a6-43d5-8eb7-92f65fa2d634/kube-rbac-proxy-metrics/0.log"
Oct 30 20:03:39.741341 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:39.741292    6777 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:40.057796 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:40.057730    6777 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"]
Oct 30 20:03:40.062175 ip-10-0-132-19 kubenswrapper[6777]: W1030 20:03:40.062139    6777 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9aea2aed_c550_4784_b822_332be0ae850e.slice/crio-b81feabbc9678ccc45906b837562b350ce6a4463cd59552213451e3da9b08e56 WatchSource:0}: Error finding container b81feabbc9678ccc45906b837562b350ce6a4463cd59552213451e3da9b08e56: Status 404 returned error can't find the container with id b81feabbc9678ccc45906b837562b350ce6a4463cd59552213451e3da9b08e56
Oct 30 20:03:40.115050 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:40.115018    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz" event={"ID":"9aea2aed-c550-4784-b822-332be0ae850e","Type":"ContainerStarted","Data":"b81feabbc9678ccc45906b837562b350ce6a4463cd59552213451e3da9b08e56"}
Oct 30 20:03:41.121673 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:41.121632    6777 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz" event={"ID":"9aea2aed-c550-4784-b822-332be0ae850e","Type":"ContainerStarted","Data":"8d64cecb0c12f618d107fd809c4412a7359636a5910d79a8b5772bdf9a6c0cf0"}
Oct 30 20:03:41.122100 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:41.121707    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:41.145198 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:41.144931    6777 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz" podStartSLOduration=2.144910588 podStartE2EDuration="2.144910588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:41.141478681 +0000 UTC m=+3403.138394069" watchObservedRunningTime="2026-10-30 20:03:41.144910588 +0000 UTC m=+3403.141825974"
Oct 30 20:03:42.048541 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:42.048505    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8466c667cd-xjs89_13941430-a959-495b-8ccc-9457c5de5236/console/0.log"
Oct 30 20:03:43.643766 ip-10-0-132-19 kubenswrapper[6777]: E1030 20:03:43.643719    6777 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vtpvd" containerName="extractor"
Oct 30 20:03:44.109956 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:44.109888    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpj4h_a979a593-d7fc-4659-a932-ca61a3801f27/dns/0.log"
Oct 30 20:03:44.139313 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:44.139282    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpj4h_a979a593-d7fc-4659-a932-ca61a3801f27/kube-rbac-proxy/0.log"
Oct 30 20:03:44.174214 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:44.174174    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-829qt_38ac8869-277b-44e4-a31a-bde13751efd9/dns-node-resolver/0.log"
Oct 30 20:03:44.959513 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:44.959485    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9b5ds_0b9c7068-7163-4c61-8a7f-5ec789247fa7/node-ca/0.log"
Oct 30 20:03:47.141228 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:47.141192    6777 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-7vlzz"
Oct 30 20:03:47.830084 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:47.829279    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-678bf485d7-bnc5s_5e038566-ad43-4e4b-b314-ade25d7b389b/kube-auth-proxy/0.log"
Oct 30 20:03:48.083755 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:48.083645    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-m5mbs_ae6b2aed-dbe2-4934-8c89-00f2f35ce325/istio-proxy/0.log"
Oct 30 20:03:48.123404 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:48.123368    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-pthrg_8084f1f2-0fb7-4dbb-b255-4ffcef554e9c/istio-proxy/0.log"
Oct 30 20:03:48.237090 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:48.237035    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wz7wl_c5a9e30d-41c9-45a4-a6ce-396e5d3f8952/istio-proxy/0.log"
Oct 30 20:03:48.282918 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:48.282888    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-xlxqw_e579180c-22b8-4904-b2a3-acad28497f49/istio-proxy/0.log"
Oct 30 20:03:48.327743 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:48.327707    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-svc7x_d5d7f713-e99d-47f0-a335-759341ccd335/payload-pre-processing/0.log"
Oct 30 20:03:48.389892 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:48.389796    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 20:03:51.297810 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:51.297783    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-678bf485d7-bnc5s_5e038566-ad43-4e4b-b314-ade25d7b389b/kube-auth-proxy/0.log"
Oct 30 20:03:51.386197 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:51.386169    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m89p4_aaa77658-47ce-47ee-b0a3-ec9dc492c3b3/serve-healthcheck-canary/0.log"
Oct 30 20:03:51.444503 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:51.444461    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-m5mbs_ae6b2aed-dbe2-4934-8c89-00f2f35ce325/istio-proxy/0.log"
Oct 30 20:03:51.471514 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:51.471481    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-pthrg_8084f1f2-0fb7-4dbb-b255-4ffcef554e9c/istio-proxy/0.log"
Oct 30 20:03:51.547571 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:51.547536    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wz7wl_c5a9e30d-41c9-45a4-a6ce-396e5d3f8952/istio-proxy/0.log"
Oct 30 20:03:51.575542 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:51.575462    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-xlxqw_e579180c-22b8-4904-b2a3-acad28497f49/istio-proxy/0.log"
Oct 30 20:03:51.601010 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:51.600978    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-svc7x_d5d7f713-e99d-47f0-a335-759341ccd335/payload-pre-processing/0.log"
Oct 30 20:03:51.632508 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:51.632478    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gblqx_2bf535e3-96e0-42c7-beef-f525cf094dae/payload-processing/0.log"
Oct 30 20:03:52.140657 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:52.140619    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vtpvd_60953151-cd1e-4572-a9f1-a618fee9ef83/kube-rbac-proxy/0.log"
Oct 30 20:03:52.174921 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:52.174886    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vtpvd_60953151-cd1e-4572-a9f1-a618fee9ef83/exporter/0.log"
Oct 30 20:03:52.210010 ip-10-0-132-19 kubenswrapper[6777]: I1030 20:03:52.209976    6777 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vtpvd_60953151-cd1e-4572-a9f1-a618fee9ef83/extractor/0.log"